Compare commits

...

70 Commits

Author SHA1 Message Date
Maria Prinus
5992df0371 SERVER-83518 Remove streams_window_operator_bm test dependency 2023-11-21 13:29:39 -08:00
Trevor Guidry
7bfc730dd6 SERVER-75033 Capture core dumps from test failures on macOS
(cherry picked from commit d6072dc2c6)
2023-11-21 20:39:00 +00:00
Spencer Jackson
d8fc3a46a7 SERVER-78066 Enable feature flag for OIDC internal authorization
(cherry picked from commit db2cc86a07)
2023-11-21 03:57:36 +00:00
Erin Zhu
0ff87204ec SERVER-82889: Fix rate limiting for views 2023-11-20 23:19:35 +00:00
Adi Zaimi
ebe878ab31 SERVER-81164 Reduce test duration by sleeping less in the bg js hook
Reduce the duration of sleep() in the run_fcv_upgrade_downgrade_background.js background hook,
which is used in the fcf_upgrade_downgrade suites, which will result in tests and consequently the suite
to take an acceptable time to complete (similar to other test suites).
2023-11-20 20:59:37 +00:00
Mihai Andrei
b0c4a578f5 SERVER-79848 Unify null semantics for $_requestResumeToken and $_resumeAfter for both engines
(cherry picked from commit 022c8097ec)
2023-11-20 16:26:38 +00:00
Luke Chen
08f189fbb2 Import wiredtiger: 4d1fae350bcc5cd7aeafe5773b849778aa5dd226 from branch mongodb-7.2
ref: e3aa043c7b..4d1fae350b
for: 7.2.0-rc2

WT-11997 Revert "WT-11398 Refresh application thread snapshot when it performs eviction (#9765)"
2023-11-20 03:33:01 +00:00
Luke Chen
fc4878a46c Import wiredtiger: e3aa043c7b1a805ec79e5318dab2130198e2f7fe from branch mongodb-7.2
ref: a613341e2b..e3aa043c7b
for: 7.2.0-rc2

WT-11851 Extend existing compaction crash test with background compaction
2023-11-20 03:33:01 +00:00
Luke Chen
dcda39c85d Import wiredtiger: a613341e2bdd4558b1a3515c75c88edb6d1a4f7b from branch mongodb-7.2
ref: d409259e2b..a613341e2b
for: 7.2.0-rc2

WT-11992 Bump WiredTiger version to 11.3.0
2023-11-20 03:33:01 +00:00
Luke Chen
3445ac6f15 Import wiredtiger: d409259e2b3a6f49667a7db67890e67fe3220aa9 from branch mongodb-7.2
ref: 9a9ba13018..d409259e2b
for: 7.2.0-rc2

WT-11583 Remove long running C tests from sanitizers
2023-11-20 03:33:01 +00:00
Luke Chen
9b00268a46 Import wiredtiger: 9a9ba13018250cfc31253d3deb0c0504e6d7d926 from branch mongodb-7.2
ref: 4b37cbe4e3..9a9ba13018
for: 7.2.0-rc2

WT-11959 Don't increment cache_inmem_split statistic when split fails
2023-11-20 03:33:01 +00:00
Luke Chen
42b8ea55dd Import wiredtiger: 4b37cbe4e3992389e0bfb903ba8ca88791f6e4ca from branch mongodb-7.2
ref: 77859760c4..4b37cbe4e3
for: 7.2.0-rc2

WT-11787 Cut WiredTiger 11.2.0 release
2023-11-20 03:33:01 +00:00
Luke Chen
54b36bb8a8 Import wiredtiger: 77859760c4bb6a23f4a58b3fc5665832b72b5a45 from branch mongodb-7.2
ref: 818f80ea53..77859760c4
for: 7.2.0-rc2

WT-11615 Extend WT functional model to support checkpoints
2023-11-20 03:33:01 +00:00
Luke Chen
f544bcdc93 Import wiredtiger: 818f80ea53d3def562483403421b5e8ddf0df757 from branch mongodb-7.2
ref: d3edc778cd..818f80ea53
for: 7.2.0-rc2

WT-11953 Make sure pre-allocated log files exist at the start of every
2023-11-20 03:33:01 +00:00
Luke Chen
06cb115d89 Import wiredtiger: d3edc778cd1637240fc807c0922563cfeb3f8c8b from branch mongodb-7.2
ref: 92cffa566b..d3edc778cd
for: 7.2.0-rc2

WT-11970 Do not raise an error if background compaction is already enabled/disabled
2023-11-20 03:33:01 +00:00
Luke Chen
df5e058182 Import wiredtiger: 92cffa566bbd5a2979a4603f598af8c1ad42e9a8 from branch mongodb-7.2
ref: 3172dc6a05..92cffa566b
for: 7.2.0-rc2

WT-11724 Have the block manager retry chunk cache reads on checksum failure
2023-11-20 03:33:01 +00:00
Luke Chen
111e48d781 Import wiredtiger: 3172dc6a05fe902f2f9ce54a85ffc2c7797906d1 from branch mongodb-7.2
ref: da214af22b..3172dc6a05
for: 7.2.0-rc2

WT-11986 Fix missing script issue for the doc-update task
2023-11-20 03:33:01 +00:00
Luke Chen
fda88f523f Import wiredtiger: da214af22bbbbf5c4c83230a1967b18b087c582a from branch mongodb-7.2
ref: 95aad3b92e..da214af22b
for: 7.2.0-rc2

WT-9628 Mark the targeted btree as dirty to avoid forcing checkpoints when compacting
2023-11-20 03:33:01 +00:00
Luke Chen
169219d7b1 Import wiredtiger: 95aad3b92eae32dfe7a8302c387acbed633f94b4 from branch mongodb-7.2
ref: c25aa02acb..95aad3b92e
for: 7.2.0-rc2

WT-11971 Fix doc-update task and artifacts archiving
2023-11-20 03:33:01 +00:00
Varun Ravichandran
0b7c20a5c1 SERVER-82143: Allow clientID to be optionally omitted during identity provider configuration 2023-11-19 04:07:50 +00:00
Gil Alon
a9d61291eb SERVER-82447 Restrict project and addFields pushdown before unpack if an expression depends on the whole document
(cherry picked from commit 0d000e6617)
2023-11-17 20:45:29 +00:00
Alexander Ignatyev
387dd839a5 SERVER-83390 Disable Boolean simplifier 2023-11-17 20:05:08 +00:00
Haley Connelly
82801a0852 SERVER-81978 handle StorageUnavailableExceptions in repl::applyCommand_inlock()
(cherry picked from commit bc2e4b24b5)
2023-11-17 17:43:30 +00:00
Alice Doherty
a2290d4a5a SERVER-82708 [v7.2] Update variants used to performance test stable branches 2023-11-17 15:02:23 +00:00
Sara Golemon
35776226b7 SERVER-83141 Add TransportLayer API for stopping new inbound connections
(cherry picked from commit 3e2542d513)
(cherry picked from commit 8c4f73c405)
2023-11-16 21:07:06 +00:00
Louis Williams
2b2d11bae3 SERVER-82305 dbCheck should ignore prepare conflicts on secondaries
(cherry picked from commit 6007ed70a6)
2023-11-16 19:51:58 +00:00
Gregory Noma
fbb3c85865 SERVER-83336 Disable wt_size_storer_cleanup_replica_set.js on macOS
(cherry picked from commit 259698664f)
2023-11-16 18:48:27 +00:00
egol
b57eb500c7 SERVER-82877 shardsvrCreateCollection request should have the same field's type as in 7.0 binary 2023-11-16 17:30:57 +00:00
Gregory Noma
7c9b028298 SERVER-33494 Remove size storer entry upon ident drop
(cherry picked from commit b4e5daaeb5)
2023-11-16 16:46:05 +00:00
Ian Boros
63e5c1ef09 SERVER-83273 Fix typo use-after-free bug in MakeObjSpec
(cherry picked from commit 4907aca9d4c9b43a45daddc387d8b64d46e431f0)
2023-11-16 14:39:51 +00:00
Aitor Esteve Alvarado
3b7e775ac5 SERVER-82685 Increase reshardingCriticalSectionTimeoutMillis for store_historical_placement_data.js
(cherry picked from commit 2d315c2035)
2023-11-16 11:05:39 +00:00
Allison Easton
12788d46a1 SERVER-81181 Enable featureFlagCheckForDirectShardOperations
(cherry picked from commit a4405631c2)
2023-11-16 08:37:55 +00:00
Erwin Pe
72e7ec8e6b SERVER-82984 Fix nModified value for queryable encrypted upserts
(cherry picked from commit d0c3d8916e)
2023-11-16 00:53:15 +00:00
Trevor Guidry
f0c87652c3 SERVER-83209 core_analyzer override rhel80-medium large distros
(cherry picked from commit cf162c1bae)
2023-11-15 09:14:07 +00:00
Trevor Guidry
197f3769aa SERVER-83206 Fix otel python deps on s390x, ppc, and macos
(cherry picked from commit 97fd77b7c1)
2023-11-15 05:06:10 +00:00
Luke Chen
81e25d61e7 Import wiredtiger: c25aa02acb695dc2da594ec92d21d4e55c9d7b15 from branch mongodb-7.2
ref: f21876bf42..c25aa02acb
for: 7.2.0-rc2

WT-11897 Split the long running tests into two test tasks
2023-11-15 02:04:06 +00:00
Luke Chen
73f59e1e33 Import wiredtiger: f21876bf42175bbeddcf24be877f6c3643caf3b8 from branch mongodb-7.2
ref: babf6984fb..f21876bf42
for: 7.2.0-rc2

WT-11885 Apply Github Apps for the doc-update task
2023-11-15 02:04:06 +00:00
Luke Chen
465b7573fa Import wiredtiger: babf6984fb12eeb800a939091a0ddc3929d2c47e from branch mongodb-7.2
ref: 9fabeef165..babf6984fb
for: 7.2.0-rc2

WT-11958 Fix the test test_app_thread_evict01 and turn on
2023-11-15 02:04:06 +00:00
Luke Chen
00842fd435 Import wiredtiger: 9fabeef165a9b81ad68ed44ef3a313e329773acf from branch mongodb-7.2
ref: f84cde5b5a..9fabeef165
for: 7.2.0-rc2

WT-11888 Fix Evergreen Python version detection on macOS
2023-11-15 02:04:06 +00:00
Luke Chen
122a1fc68f Import wiredtiger: f84cde5b5a24e543d0504e76b1cf3c0d777e6035 from branch mongodb-7.2
ref: de98396412..f84cde5b5a
for: 7.2.0-rc2

WT-11778 Restricting dirty eviction to sessions with snapshot isolation and of metadata pages
2023-11-15 02:04:06 +00:00
Luke Chen
7bf9b19da0 Import wiredtiger: de98396412fd5ce1084da888cb742a54e4dda9fe from branch mongodb-7.2
ref: a7926f926d..de98396412
for: 7.2.0-rc2

WT-11951 Fix prefetching integration with verify operation
2023-11-15 02:04:06 +00:00
Luke Chen
06ef441cc9 Import wiredtiger: a7926f926d88653c8e6698e87a56d6f8db7b52bb from branch mongodb-7.2
ref: d50915cc93..a7926f926d
for: 7.2.0-rc2

WT-9287 Add auto loading to GDB functions
2023-11-15 02:04:06 +00:00
Luke Chen
0852bf5837 Import wiredtiger: d50915cc93cd19d5f74fb92ad79403011c28debf from branch mongodb-7.2
ref: 4e00d59d01..d50915cc93
for: 7.2.0-rc2

WT-11793 Remove uses of WT_ internal macros in cpp suite: WT_BILLION,
2023-11-15 02:04:06 +00:00
Luke Chen
7830295d37 Import wiredtiger: 4e00d59d01598826f64478429e52bc24f2dbd68c from branch mongodb-7.2
ref: 8d844cb8d9..4e00d59d01
for: 7.2.0-rc2

WT-11955 Fix list formating bug in CONTRIBUTING.rst
2023-11-15 02:04:06 +00:00
Luke Chen
f11d8d3ffd Import wiredtiger: 8d844cb8d9572f3ea0bb9c69ddd9d5a1a7663cc0 from branch mongodb-7.2
ref: 578574b2c7..8d844cb8d9
for: 7.2.0-rc2

WT-11614 Parse the debug log into a set of corresponding functional model operations
2023-11-15 02:04:06 +00:00
Luke Chen
d86a8f3aed Import wiredtiger: 578574b2c7ac91bf859e30fd04c588e0cdf29523 from branch mongodb-7.2
ref: f9183b017d..578574b2c7
for: 7.2.0-rc2

WT-11217 Callback based session array traversal
2023-11-15 02:04:06 +00:00
Luke Chen
4f13c63a87 Import wiredtiger: f9183b017d6e455c6f97dcf7244b2ec0040dc1eb from branch mongodb-7.2
ref: a6340032e1..f9183b017d
for: 7.2.0-rc2

WT-9607 Fix timing of compact progress messages
2023-11-15 02:04:06 +00:00
Luke Chen
cf9545f43e Import wiredtiger: a6340032e1e1770fec1b493f0dd344458a28fee6 from branch mongodb-7.2
ref: 246e15a725..a6340032e1
for: 7.2.0-rc2

WT-11938 Allow Evergreen to apply options after ENABLE_TCMALLOC
2023-11-15 02:04:06 +00:00
Luke Chen
1e0bc6c19d Import wiredtiger: 246e15a72598acbcfd8775d46083205af543744f from branch mongodb-7.2
ref: 46e5e87098..246e15a725
for: 7.2.0-rc2

WT-11909 Add missing compile step for ex_extra_diagnostics
2023-11-15 02:04:06 +00:00
Kevin Cherkauer
e5ed10c76e SERVER-82736 Fix sbe::UnwindStage yield handling to not use freed memory 2023-11-14 21:09:00 +00:00
roganamy
9c6df663e5 SERVER-82954: Fix for BF-30717
(cherry picked from commit 557d12b8bd)
2023-11-14 18:15:13 +00:00
SimonEismann
79d63dc704 SERVER-82351 Revert "SERVER-80723: Clean up system_perf.yml" 2023-11-14 12:57:59 +00:00
Mickey. J Winters
0592edb56b SERVER-83150 make Document::shred() copy document metadata 2023-11-14 04:36:32 +00:00
Huayu Ouyang
3884d84f29 SERVER-79274 Fix race where FCV is uninitialized in between FCV checks
(cherry picked from commit a5da6271d5)
2023-11-14 03:30:12 +00:00
Erin Zhu
174ec4ffea SERVER-82782 Uses ISODate for "asOf" in $queryStats
(cherry picked from commit 6d70ea8342)
2023-11-13 23:45:53 +00:00
Dan Larkin-York
cf1865b616 SERVER-82090 Reduce scope of synchronization for archive-based reopening
(cherry picked from commit e98f26d0f9)
2023-11-13 23:03:19 +00:00
Uladzimir Makouski
b83466d202 Revert "SERVER-82877 shardsvrCreateCollection request should have the same field's type as in 7.0 binary"
This reverts commit b8d09cc1fe.
2023-11-11 01:17:41 +00:00
egol
b8d09cc1fe SERVER-82877 shardsvrCreateCollection request should have the same field's type as in 7.0 binary 2023-11-10 17:59:18 +00:00
Tommaso Tocci
215a1ef0d2 SERVER-83005 Reintroduce numInitialChunks parameter (IDL only) to maintain backward compatibility 2023-11-09 16:44:24 +00:00
Jordi Serra Torrens
c112a6acd1 SERVER-82361 Do not swallow StorageUnavailable if within a wuow
(cherry picked from commit 9a34542a8c)
2023-11-09 12:47:10 +00:00
Kruti Shah
0cfe182cb7 SERVER-82799 Increase reshardingCriticalSectionTimeoutMillis for resharding_update_tag_zones.js
(cherry picked from commit d6548865b5)
2023-11-08 18:16:47 +00:00
Mikhail Shchatko
c3f1e65a17 SERVER-82133 Copy required variants to etc/evergreen_nightly.yml 2023-11-07 18:36:31 +00:00
Mikhail Shchatko
7d9af0f285 SERVER-82133 Remove variants with "-wtdevelop" suffix from etc/evergreen_yml_components/variants/compile_static_analysis.yml 2023-11-07 18:36:31 +00:00
Mikhail Shchatko
c69d1393e4 SERVER-82133 Remove future git tag compile variant from etc/evergreen_yml_components/variants/compile_static_analysis.yml 2023-11-07 18:36:31 +00:00
Mikhail Shchatko
816b276820 SERVER-82133 Remove the FCV sys-perf variants from etc/system_perf.yml 2023-11-07 18:36:31 +00:00
Mikhail Shchatko
7d436a49a7 SERVER-82133 Remove all feature flags variants from etc/system_perf.yml 2023-11-07 18:36:31 +00:00
Mikhail Shchatko
c490817b9f SERVER-82133 Remove renew_ssl_cert from etc/system_perf.yml 2023-11-07 18:36:31 +00:00
Mikhail Shchatko
33735658d9 SERVER-82133 Change enterprise module branch name from "master" to "v7.2" in etc/system_perf.yml 2023-11-07 18:36:31 +00:00
Mikhail Shchatko
84bba029b1 SERVER-82133 Update filename suffixes in buildscripts/generate_version_expansions.py from "latest" to "v7.2-latest" 2023-11-07 18:36:31 +00:00
Mikhail Shchatko
a3c3ecf14d SERVER-82133 Change references from "master" to "v7.2" in etc/evergreen_yml_components/definitions.yml 2023-11-07 18:36:31 +00:00
363 changed files with 7322 additions and 4695 deletions

View File

@ -61,8 +61,8 @@ def generate_version_expansions():
raise ValueError("Unable to parse version from stdin and no version.json provided")
if version_parts[0]:
expansions["suffix"] = "latest"
expansions["src_suffix"] = "latest"
expansions["suffix"] = "v7.2-latest"
expansions["src_suffix"] = "v7.2-latest"
expansions["is_release"] = "false"
else:
expansions["suffix"] = version_line

View File

@ -20,6 +20,7 @@ GENERIC_FCV = [
r'::kDowngradingFromLatestToLastLTS',
r'::kDowngradingFromLatestToLastContinuous',
r'\.isUpgradingOrDowngrading',
r'->isUpgradingOrDowngrading',
r'::kDowngradingFromLatestToLastContinuous',
r'::kUpgradingFromLastLTSToLastContinuous',
]

View File

@ -7,7 +7,6 @@ selector:
# The trailing asterisk is for handling the .exe extension on Windows.
- build/**/system_resource_canary_bm*
- build/install/bin/streams_operator_dag_bm*
- build/install/bin/streams_window_operator_bm*
executor:
config: {}

View File

@ -101,8 +101,11 @@ def generate(expansions_file: str = "../expansions.yml",
# gather information from the current task being run
expansions = read_config_file(expansions_file)
distro = expansions.get(
"large_distro_name") if "large_distro_name" in expansions else expansions.get("distro_id")
distro = None
for distro_expansion in ["core_analyzer_distro_name", "large_distro_name", "distro_id"]:
if distro := expansions.get(distro_expansion, None):
break
assert distro is not None
current_task_name = expansions.get("task_name")
task_id = expansions.get("task_id")
execution = expansions.get("execution")

View File

@ -495,6 +495,22 @@ last-continuous:
ticket: SERVER-81985
- test_file: jstests/core/timeseries/timeseries_collection_uuid.js
ticket: SERVER-81379
- test_file: src/mongo/db/modules/enterprise/jstests/fle2/basic_update.js
ticket: SERVER-82984
- test_file: src/mongo/db/modules/enterprise/jstests/fle2/basic_replace.js
ticket: SERVER-82984
- test_file: jstests/core/timeseries/timeseries_project_pushdown.js
ticket: SERVER-82447
- test_file: jstests/sharding/resharding_secondary_recovers_temp_ns_metadata.js
ticket: SERVER-79848
- test_file: jstests/core/find_with_resume_after_param.js
ticket: SERVER-79848
- test_file: jstests/core/timeseries/timeseries_resume_after.js
ticket: SERVER-79848
- test_file: src/mongo/db/modules/enterprise/jstests/external_auth_oidc/oidc_internal_authz.js
ticket: SERVER-78066
- test_file: src/mongo/db/modules/enterprise/jstests/external_auth_oidc/oidc_omit_client_id.js
ticket: SERVER-78066
suites: null
last-lts:
all:
@ -1052,4 +1068,16 @@ last-lts:
ticket: SERVER-81985
- test_file: jstests/core/timeseries/timeseries_collection_uuid.js
ticket: SERVER-81379
- test_file: src/mongo/db/modules/enterprise/jstests/fle2/basic_update.js
ticket: SERVER-82984
- test_file: src/mongo/db/modules/enterprise/jstests/fle2/basic_replace.js
ticket: SERVER-82984
- test_file: jstests/core/timeseries/timeseries_project_pushdown.js
ticket: SERVER-82447
- test_file: jstests/sharding/resharding_secondary_recovers_temp_ns_metadata.js
ticket: SERVER-79848
- test_file: jstests/core/find_with_resume_after_param.js
ticket: SERVER-79848
- test_file: jstests/core/timeseries/timeseries_resume_after.js
ticket: SERVER-79848
suites: null

View File

@ -78,15 +78,15 @@ variables:
- name: generate_buildid_to_debug_symbols_mapping
variant: linux-x86-dynamic-compile
- &linux_x86_compile_variant_dependency
- &amazon2_x86_compile_variant_dependency
depends_on:
- name: archive_dist_test_debug
variant: &linux_x86_compile_variant_name linux-x86-compile #static compile
variant: &amazon2_x86_compile_variant_name amazon2-x86-compile #static compile
- name: version_gen
variant: generate-tasks-for-version
omit_generated_tasks: true
- name: generate_buildid_to_debug_symbols_mapping
variant: linux-x86-compile
variant: amazon2-x86-compile
- &amazon_linux2_arm64_dynamic_compile_variant_dependency
depends_on:
@ -152,15 +152,16 @@ variables:
multiversion_edition: enterprise
repo_edition: enterprise
large_distro_name: rhel80-medium
core_analyzer_distro_name: rhel80-large
num_scons_link_jobs_available: 0.99
compile_variant: *linux_x86_dynamic_compile_variant_name
- &linux_x86_expansions
- &amazon2_x86_expansions
multiversion_platform: amazon2
multiversion_edition: enterprise
repo_edition: enterprise
num_scons_link_jobs_available: 0.99
compile_variant: *linux_x86_compile_variant_name
compile_variant: *amazon2_x86_compile_variant_name
- &linux_arm64_generic_expansions
multiversion_platform: amazon2
@ -399,14 +400,14 @@ buildvariants:
###########################################
# Linux buildvariants #
###########################################
- <<: *linux_x86_compile_variant_dependency
- <<: *amazon2_x86_compile_variant_dependency
name: perf-standalone-classic-query-engine.intel.aws.2023-09
display_name: PERF Standalone (Classic Query Engine) Intel AWS 2023-09
# cron: &cron_query_perf_schedule "0 0 * * 4" # 00:00 on Thursday
activate: false
modules: *perf_modules
expansions:
<<: *linux_x86_expansions
<<: *amazon2_x86_expansions
idle_timeout_factor: 1.5
exec_timeout_factor: 1.5
mongodb_setup_release: 2022-11
@ -418,7 +419,7 @@ buildvariants:
project_dir: *perf_project_dir
authentication: enabled
storageEngine: wiredTiger
compile_variant: *linux_x86_compile_variant_name
compile_variant: *amazon2_x86_compile_variant_name
run_on:
- amazon2
tasks: &classic_engine_tasks
@ -456,6 +457,7 @@ buildvariants:
target_resmoke_time: 15
max_sub_suites: 5
large_distro_name: rhel80-medium
core_analyzer_distro_name: rhel80-large
compile_variant: *linux_x86_dynamic_debug_compile_variant_name
tasks:
- name: .aggregation !.encrypt !.feature_flag_guarded
@ -1402,9 +1404,8 @@ buildvariants:
- name: .causally_consistent !.sharding
- name: .change_stream_fuzzer
- name: .change_streams
# TODO(SERVER-79848): Remove !.sharded
- name: .concurrency !.large !.ubsan !.no_txns !.compute_mode !.sharded
- name: .concurrency .large !.ubsan !.no_txns !.compute_mode !.sharded
- name: .concurrency !.large !.ubsan !.no_txns !.compute_mode
- name: .concurrency .large !.ubsan !.no_txns !.compute_mode
distros:
- rhel80-medium
- name: .encrypt
@ -2084,8 +2085,7 @@ buildvariants:
- name: .change_streams
# - name: disk_wiredtiger
- name: .misc_js
# TODO(SERVER-79848): Remove !.sharded
- name: .concurrency !.ubsan !.no_txns !.kill_terminate !.compute_mode !.sharded
- name: .concurrency !.ubsan !.no_txns !.kill_terminate !.compute_mode
- name: .encrypt
- name: external_auth
- name: external_auth_aws
@ -2104,8 +2104,7 @@ buildvariants:
- name: replica_sets_large_txns_format_jscore_passthrough
- name: .replica_sets !.multi_oplog
- name: .replica_sets .encrypt
# TODO(SERVER-79848): Re-enable resharding_fuzzer
# - name: .resharding_fuzzer
- name: .resharding_fuzzer
- name: .retry
- name: .rollbackfuzzer
- name: .read_only
@ -2249,6 +2248,7 @@ buildvariants:
scons_cache_scope: shared
separate_debug: off
large_distro_name: rhel80-medium
core_analyzer_distro_name: rhel80-large
multiversion_platform: rhel80
multiversion_edition: enterprise
test_flags: >-
@ -3000,9 +3000,8 @@ buildvariants:
- name: change_streams_multitenant_sharded_collections_passthrough
- name: .misc_js
- name: .clustered_collections
# TODO(SERVER-79848): Remove !.sharded
- name: .concurrency !.large !.ubsan !.no_txns !.sharded
- name: .concurrency .large !.ubsan !.no_txns !.sharded
- name: .concurrency !.large !.ubsan !.no_txns
- name: .concurrency .large !.ubsan !.no_txns
distros:
- rhel80-medium
- name: .config_fuzzer !.large

View File

@ -6,20 +6,565 @@ include:
- filename: etc/evergreen_yml_components/variants/atlas.yml
- filename: etc/evergreen_yml_components/variants/misc_release.yml
### Comment out when using this file for a Rapid release branch. ###
- filename: etc/evergreen_yml_components/variants/ibm.yml
# - filename: etc/evergreen_yml_components/variants/ibm.yml
### Uncomment when using this file for a LTS release branch. ###
# - filename: etc/evergreen_yml_components/variants/in_memory.yml
### Uncomment when using this file for a LTS or Rapid release branch. ###
# - filename: etc/evergreen_yml_components/variants/sanitizer.yml
- filename: etc/evergreen_yml_components/variants/sanitizer.yml
### Uncomment when using this file for a LTS or Rapid release branch. ###
# - filename: etc/evergreen_yml_components/variants/ninja.yml
- filename: etc/evergreen_yml_components/variants/ninja.yml
### Uncomment when using this file for a LTS or Rapid release branch. ###
# - filename: etc/evergreen_yml_components/variants/classic_engine.yml
- filename: etc/evergreen_yml_components/variants/classic_engine.yml
### Uncomment when using this file for a LTS or Rapid release branch. ###
# - filename: etc/evergreen_yml_components/variants/config_shard.yml
- filename: etc/evergreen_yml_components/variants/config_shard.yml
- filename: etc/evergreen_yml_components/variants/compile_static_analysis.yml
parameters:
- key: evergreen_config_file_path
value: "etc/evergreen_nightly.yml"
description: "path to this file"
variables:
- &linux_x86_dynamic_debug_compile_variant_dependency
depends_on:
- name: archive_dist_test_debug
variant: &linux_x86_dynamic_debug_compile_variant_name linux-x86-dynamic-debug-compile-required
- name: version_gen
variant: generate-tasks-for-version
# This is added because of EVG-18211.
# Without this we are adding extra dependencies on evergreen and it is causing strain
omit_generated_tasks: true
- name: generate_buildid_to_debug_symbols_mapping
variant: linux-x86-dynamic-debug-compile-required
- &windows_compile_variant_dependency
depends_on:
- name: archive_dist_test_debug
variant: &windows_compile_variant_name windows-compile-required
- name: version_gen
variant: generate-tasks-for-version
# This is added because of EVG-18211.
# Without this we are adding extra dependencies on evergreen and it is causing strain
omit_generated_tasks: true
- name: generate_buildid_to_debug_symbols_mapping
variant: windows-compile-required
- &amazon_linux2_arm64_dynamic_compile_variant_dependency
depends_on:
- name: archive_dist_test_debug
variant: &amazon_linux2_arm64_dynamic_compile_variant_name amazon-linux2-arm64-dynamic-compile
- name: version_gen
variant: generate-tasks-for-version
# This is added because of EVG-18211.
# Without this we are adding extra dependencies on evergreen and it is causing strain
omit_generated_tasks: true
- name: generate_buildid_to_debug_symbols_mapping
variant: amazon-linux2-arm64-dynamic-compile
# Common compile variant dependency specifications.
# THIS WAS COPIED TO config_shard.yml - ANY MODIFICATIONS HERE SHOULD ALSO BE MADE IN THAT FILE.
- &linux_x86_dynamic_compile_variant_dependency
depends_on:
- name: archive_dist_test_debug
variant: &linux_x86_dynamic_compile_variant_name linux-x86-dynamic-compile
- name: version_gen
variant: generate-tasks-for-version
# This is added because of EVG-18211.
# Without this we are adding extra dependencies on evergreen and it is causing strain
omit_generated_tasks: true
- name: generate_buildid_to_debug_symbols_mapping
variant: linux-x86-dynamic-compile
# THIS WAS COPIED TO config_shard.yml - ANY MODIFICATIONS HERE SHOULD ALSO BE MADE IN THAT FILE.
- &linux_x86_generic_expansions
multiversion_platform: rhel80
multiversion_edition: enterprise
repo_edition: enterprise
large_distro_name: rhel80-medium
num_scons_link_jobs_available: 0.99
compile_variant: *linux_x86_dynamic_compile_variant_name
- &linux_debug_aubsan_compile_variant_dependency
depends_on:
- name: archive_dist_test_debug
variant: &linux_debug_aubsan_compile_variant_name linux-debug-aubsan-compile-required
- name: version_gen
variant: generate-tasks-for-version
# This is added because of EVG-18211.
# Without this we are adding extra dependencies on evergreen and it is causing strain
omit_generated_tasks: true
- name: generate_buildid_to_debug_symbols_mapping
variant: linux-debug-aubsan-compile-required
# If you add anything to san_options, make sure the appropriate changes are
# also made to SConstruct.
# and also to the san_options in compile_static_analysis.yml and sanitizer.yml
- aubsan_options: &aubsan_options
>-
UBSAN_OPTIONS="print_stacktrace=1:external_symbolizer_path=/opt/mongodbtoolchain/v4/bin/llvm-symbolizer"
LSAN_OPTIONS="suppressions=etc/lsan.suppressions:report_objects=1"
ASAN_OPTIONS="detect_leaks=1:check_initialization_order=true:strict_init_order=true:abort_on_error=1:disable_coredump=0:handle_abort=1:strict_string_checks=true:detect_invalid_pointer_pairs=1:external_symbolizer_path=/opt/mongodbtoolchain/v4/bin/llvm-symbolizer"
- &linux_arm64_generic_expansions
multiversion_platform: amazon2
multiversion_edition: enterprise
multiversion_architecture: aarch64
packager_arch: aarch64
packager_distro: amazon2
repo_edition: enterprise
large_distro_name: amazon2-arm64-large
num_scons_link_jobs_available: 0.99
compile_variant: *amazon_linux2_arm64_dynamic_compile_variant_name
- &enterprise-amazon-linux2-arm64-all-feature-flags-template
<<: *amazon_linux2_arm64_dynamic_compile_variant_dependency
name: &enterprise-amazon-linux2-arm64-all-feature-flags enterprise-amazon-linux2-arm64-all-feature-flags
display_name: "! Amazon Linux 2 arm64 (all feature flags)"
cron: "0 */4 * * *" # From the ${project_required_suggested_cron} parameter
modules:
- enterprise
run_on:
- amazon2-arm64-small
stepback: true
expansions: &enterprise-amazon-linux2-arm64-all-feature-flags-expansions
<<: *linux_arm64_generic_expansions
scons_cache_scope: shared
scons_cache_mode: all
has_packages: false
jstestfuzz_num_generated_files: 40
jstestfuzz_concurrent_num_files: 10
target_resmoke_time: 10
max_sub_suites: 5
idle_timeout_factor: 1.5
exec_timeout_factor: 1.5
test_flags: >-
--runAllFeatureFlagTests
--excludeWithAnyTags=incompatible_with_amazon_linux,requires_external_data_source
tasks:
- name: analyze_shard_key_jscore_passthrough_gen
- name: query_golden_classic
- name: lint_fuzzer_sanity_patch
- name: test_api_version_compatibility
- name: check_feature_flag_tags
- name: check_for_todos
- name: .aggfuzzer !.cqf_only
- name: .aggregation
- name: aggregation_repeat_queries
- name: audit
- name: .auth
- name: buildscripts_test
- name: .bulk_write
- name: resmoke_end2end_tests
- name: unittest_shell_hang_analyzer_gen
- name: .causally_consistent !.sharding
- name: .change_streams
- name: .change_stream_fuzzer
# TODO SERVER-57866: Remove the explicit mentions of change stream multitenant suites.
- name: change_streams_multitenant_passthrough
- name: change_streams_multitenant_sharded_collections_passthrough
- name: .cqf
- name: .misc_js
- name: .clustered_collections
- name: .concurrency !.large !.ubsan !.no_txns
- name: .concurrency .large !.ubsan !.no_txns
- name: .config_fuzzer !.large
- name: .config_fuzzer .large
distros:
- amazon2-arm64-large
- name: .config_fuzzer_stress
distros:
- amazon2-arm64-large
- name: disk_wiredtiger
- name: .encrypt
- name: feature_flag_multiversion_gen
- name: idl_tests
- name: initial_sync_fuzzer_gen
- name: .jscore .common
- name: jsCore_column_store_indexes
- name: jsCore_min_batch_repeat_queries_ese_gsm
- name: jsCore_txns_large_txns_format
- name: jsCore_wildcard_indexes
- name: json_schema
- name: .jstestfuzz !.flow_control # Flow control jstestfuzz take longer.
- name: libunwind_tests
- name: .multiversion_sanity_check
- name: .multi_shard
- name: multi_stmt_txn_jscore_passthrough_with_migration_gen
- name: multiversion_gen
- name: powercycle_smoke
- name: .query_fuzzer !.cqf_only
- name: .random_multiversion_ds
- name: .read_write_concern .large
- name: .read_write_concern !.large
- name: .replica_sets !.encrypt !.auth
- name: replica_sets_api_version_jscore_passthrough_gen
- name: replica_sets_reconfig_jscore_passthrough_gen
- name: replica_sets_reconfig_jscore_stepdown_passthrough_gen
- name: replica_sets_reconfig_kill_primary_jscore_passthrough_gen
- name: change_streams_pre_images_replica_sets_stepdown_primary_jscore_passthrough_gen
- name: change_streams_pre_images_replica_sets_kill_secondary_jscore_passthrough_gen
- name: change_streams_change_collection_replica_sets_stepdown_primary_jscore_passthrough_gen
- name: change_streams_change_collection_replica_sets_kill_secondary_jscore_passthrough_gen
- name: retryable_writes_jscore_passthrough_gen
- name: retryable_writes_jscore_stepdown_passthrough_gen
- name: .read_only
- name: .rollbackfuzzer
- name: sasl
- name: search
- name: search_auth
- name: search_pinned_connections_auth
- name: search_ssl
- name: secondary_reads_passthrough_gen
- name: .serverless
distros:
- amazon2-arm64-large
- name: session_jscore_passthrough
- name: .shard_split
- name: .sharding .jscore !.wo_snapshot !.multi_stmt
- name: sharding_api_version_jscore_passthrough_gen
- name: sharding_api_strict_passthrough_gen
- name: .sharding .txns
- name: .sharding .common
- name: sharding_max_mirroring_opportunistic_secondary_targeting_ese_gcm_gen
- name: sharded_collections_single_writes_without_shard_key_jscore_passthrough_gen
- name: sharded_multi_stmt_txn_jscore_passthrough
- name: streams
- name: streams_auth
- name: .updatefuzzer
- name: vector_search
- name: vector_search_auth
- name: vector_search_ssl
- name: query_stats_passthrough
- name: query_stats_passthrough_writeonly
- name: query_stats_mongos_passthrough
buildvariants:
- &linux-64-debug-required-template
<<: *linux_x86_dynamic_debug_compile_variant_dependency
name: &linux-64-debug-required linux-64-debug-required
display_name: "! Linux x86 Shared Library DEBUG"
cron: "0 */4 * * *" # From the ${project_required_suggested_cron} parameter
run_on:
- rhel80-medium
expansions:
resmoke_jobs_factor: 0.5 # Avoid starting too many mongod's
test_flags: --excludeWithAnyTags=requires_http_client
target_resmoke_time: 15
max_sub_suites: 5
large_distro_name: rhel80-medium
compile_variant: *linux_x86_dynamic_debug_compile_variant_name
tasks:
- name: .aggregation !.encrypt !.feature_flag_guarded
- name: .auth !.audit !.multiversion
- name: sharding_auth_gen
- name: .causally_consistent !.wo_snapshot
- name: .change_streams
- name: .clustered_collections
- name: .misc_js
- name: disk_wiredtiger
- name: .jscore .common
- name: jsCore_txns_large_txns_format
- name: json_schema
- name: query_golden_classic
- name: libunwind_tests
- name: .multi_shard
- name: multi_stmt_txn_jscore_passthrough_with_migration_gen
- name: .ocsp
- name: .read_write_concern
- name: .replica_sets !.encrypt !.fcbis
- name: replica_sets_reconfig_jscore_passthrough_gen
- name: replica_sets_reconfig_jscore_stepdown_passthrough_gen
- name: replica_sets_max_mirroring_large_txns_format_gen
- name: .retry
- name: .read_only
- name: session_jscore_passthrough
- name: sharded_multi_stmt_txn_jscore_passthrough
- name: .sharding .jscore !.wo_snapshot
- name: sharding_gen
- name: sharding_max_mirroring_opportunistic_secondary_targeting_gen
- &enterprise-windows-template
<<: *windows_compile_variant_dependency
name: &enterprise-windows-required enterprise-windows-required
display_name: "! Enterprise Windows"
cron: "0 */4 * * *" # From the ${project_required_suggested_cron} parameter
modules:
- enterprise
run_on:
- windows-vsCurrent-small
expansions: &windows_required_expansions
compile_variant: *windows_compile_variant_name
burn_in_tests_build_variant: enterprise-windows-required
exe: ".exe"
content_type: application/zip
python: '/cygdrive/c/python/python39/python.exe'
ext: zip
multiversion_platform: windows
multiversion_edition: enterprise
jstestfuzz_num_generated_files: 35
target_resmoke_time: 20
max_sub_suites: 5
large_distro_name: windows-vsCurrent-large
push_path: windows
push_bucket: downloads.10gen.com
push_name: windows
push_arch: x86_64-enterprise
test_flags: &windows_common_test_excludes --excludeWithAnyTags=incompatible_with_windows_tls
external_auth_jobs_max: 1
tasks:
- name: audit
- name: auth_audit_gen
- name: causally_consistent_jscore_txns_passthrough
distros:
- windows-vsCurrent-large
- name: .encrypt !.aggregation !.replica_sets !.sharding !.jscore
- name: external_auth
- name: external_auth_aws
- name: external_auth_windows
distros:
- windows-2016-dc
- name: .jscore .common !.sharding
- name: jsCore_auth
- name: jsCore_ese
- name: jsCore_txns_large_txns_format
- name: .jstestfuzz .common
- name: noPassthrough_gen
- name: noPassthroughWithMongod_gen
- name: .replica_sets .common
- name: .replica_sets .multi_oplog !.gcm
- name: sasl
- name: .sharding .txns
- name: sharding_auth_audit_gen
- name: sharding_max_mirroring_opportunistic_secondary_targeting_ese_gen
- name: run-all-affected-jstests
display_name: "! Run All Affected JStests"
patch_only: true
run_on:
- rhel80-medium
expansions:
large_distro_name: rhel80-large
burn_in_tag_include_all_required_and_suggested: true
burn_in_tag_exclude_build_variants: >-
macos-debug-suggested
burn_in_tag_include_build_variants:
burn_in_tag_compile_task_dependency: archive_dist_test_debug
compile_variant: *amazon_linux2_arm64_dynamic_compile_variant_name
depends_on:
- name: archive_dist_test_debug
variant: *amazon_linux2_arm64_dynamic_compile_variant_name
- name: version_gen
variant: generate-tasks-for-version
# This is added because of EVG-18211.
# Without this we are adding extra dependencies on evergreen and it is causing strain
omit_generated_tasks: true
- name: version_burn_in_gen
variant: generate-tasks-for-version
# This is added because of EVG-18211.
# Without this we are adding extra dependencies on evergreen and it is causing strain
omit_generated_tasks: true
tasks:
- name: burn_in_tags_gen
- &enterprise-rhel-80-64-bit-dynamic-template
<<: *linux_x86_dynamic_compile_variant_dependency
name: &enterprise-rhel-80-64-bit-dynamic enterprise-rhel-80-64-bit-dynamic
display_name: "! Shared Library Enterprise RHEL 8.0"
cron: "0 */4 * * *" # From the ${project_required_suggested_cron} parameter
modules:
- enterprise
run_on:
- rhel80-small
# THIS WAS COPIED TO config_shard.yml - ANY MODIFICATIONS HERE SHOULD ALSO BE MADE IN THAT FILE.
expansions: &enterprise-rhel-80-64-bit-dynamic-expansions
<<: *linux_x86_generic_expansions
scons_cache_scope: shared
scons_cache_mode: all
has_packages: false
jstestfuzz_num_generated_files: 40
jstestfuzz_concurrent_num_files: 10
target_resmoke_time: 10
max_sub_suites: 5
idle_timeout_factor: 1.5
exec_timeout_factor: 1.5
large_distro_name: rhel80-medium
depends_on:
- name: archive_dist_test_debug
variant: *linux_x86_dynamic_compile_variant_name
- name: version_gen
variant: generate-tasks-for-version
# This is added because of EVG-18211.
# Without this we are adding extra dependencies on evergreen and it is causing strain
omit_generated_tasks: true
tasks:
- name: .aggfuzzer !.feature_flag_guarded
- name: .aggregation !.feature_flag_guarded
- name: aggregation_repeat_queries
- name: analyze_shard_key_jscore_passthrough_gen
- name: audit
- name: .auth
- name: unittest_shell_hang_analyzer_gen
- name: .causally_consistent !.sharding
- name: .change_streams
- name: .change_stream_fuzzer
- name: .misc_js
- name: .concurrency !.large !.ubsan !.no_txns
- name: .concurrency .large !.ubsan !.no_txns
distros:
- rhel80-medium
- name: .config_fuzzer !.large
- name: .config_fuzzer .large
distros:
- rhel80-medium
- name: disk_wiredtiger
- name: .encrypt
- name: idl_tests
- name: initial_sync_fuzzer_gen
- name: fcv_upgrade_downgrade_replica_sets_jscore_passthrough_gen
- name: fcv_upgrade_downgrade_sharding_jscore_passthrough_gen
- name: fcv_upgrade_downgrade_sharded_collections_jscore_passthrough_gen
- name: jsCore
distros:
- rhel80-xlarge
- name: .jscore .common !jsCore
- name: jsCore_min_batch_repeat_queries_ese_gsm
- name: jsCore_txns_large_txns_format
- name: json_schema
- name: .jstestfuzz !.flow_control # Flow control jstestfuzz take longer.
- name: libunwind_tests
- name: .multi_shard
- name: multi_stmt_txn_jscore_passthrough_with_migration_gen
- name: multiversion_gen
- name: .query_fuzzer !.feature_flag_guarded
- name: .random_multiversion_ds
- name: .read_write_concern .large
distros:
- rhel80-medium
- name: .read_write_concern !.large
- name: .replica_sets !.encrypt !.auth
distros:
- rhel80-xlarge
- name: replica_sets_api_version_jscore_passthrough_gen
- name: replica_sets_reconfig_jscore_passthrough_gen
- name: replica_sets_reconfig_jscore_stepdown_passthrough_gen
distros:
- rhel80-xlarge
- name: replica_sets_reconfig_kill_primary_jscore_passthrough_gen
distros:
- rhel80-xlarge
- name: change_streams_pre_images_replica_sets_stepdown_primary_jscore_passthrough_gen
distros:
- rhel80-xlarge
- name: change_streams_pre_images_replica_sets_kill_secondary_jscore_passthrough_gen
distros:
- rhel80-xlarge
- name: change_streams_change_collection_replica_sets_stepdown_primary_jscore_passthrough_gen
distros:
- rhel80-xlarge
- name: change_streams_change_collection_replica_sets_kill_secondary_jscore_passthrough_gen
distros:
- rhel80-xlarge
- name: retryable_writes_jscore_passthrough_gen
- name: retryable_writes_jscore_stepdown_passthrough_gen
distros:
- rhel80-medium
- name: .read_only
- name: .rollbackfuzzer
- name: sasl
- name: search
- name: search_auth
- name: search_pinned_connections_auth
- name: search_ssl
- name: session_jscore_passthrough
- name: sharded_collections_single_writes_without_shard_key_jscore_passthrough_gen
- name: .sharding .jscore !.wo_snapshot !.multi_stmt
- name: sharding_api_version_jscore_passthrough_gen
- name: sharding_api_strict_passthrough_gen
- name: .sharding .txns
- name: .sharding .common
- name: .updatefuzzer
- name: secondary_reads_passthrough_gen
- name: .serverless
distros:
- rhel80-xlarge
- name: vector_search
- name: vector_search_auth
- name: vector_search_ssl
- &rhel80-debug-aubsan-lite-template
<<: *linux_debug_aubsan_compile_variant_dependency
name: &rhel80-debug-aubsan-lite rhel80-debug-aubsan-lite
display_name: "! Shared Library {A,UB}SAN Enterprise RHEL 8.0 DEBUG"
cron: "0 */4 * * *" # From the ${project_required_suggested_cron} parameter
modules:
- enterprise
run_on:
- rhel80-build
expansions: &aubsan-lite-required-expansions
compile_variant: *linux_debug_aubsan_compile_variant_name
lang_environment: LANG=C
san_options: *aubsan_options
test_flags: --excludeWithAnyTags=requires_ocsp_stapling,requires_increased_memlock_limits
resmoke_jobs_factor: 0.3 # Avoid starting too many mongod's under {A,UB}SAN build.
hang_analyzer_dump_core: false
max_sub_suites: 3
num_scons_link_jobs_available: 0.99
large_distro_name: rhel80-build
multiversion_platform: rhel80
multiversion_edition: enterprise
gcov_tool: /opt/mongodbtoolchain/v4/bin/gcov
tasks:
- name: jsCore
- name: jsCore_txns
- <<: *enterprise-amazon-linux2-arm64-all-feature-flags-template
name: &commit-queue commit-queue
display_name: "~ Commit Queue"
cron: "0 4 * * 0" # From the ${project_weekly_cron} parameter
stepback: false
expansions:
<<: *linux_arm64_generic_expansions
scons_cache_scope: shared
scons_cache_mode: all
commit_queue_alternate_cache: amazon-linux2-arm64-compile
has_packages: false
compile_flags: >-
--ssl
MONGO_DISTMOD=amazon2
-j$(grep -c ^processor /proc/cpuinfo)
--variables-files=etc/scons/mongodbtoolchain_stable_gcc.vars
--link-model=dynamic
crypt_task_compile_flags: >-
SHLINKFLAGS_EXTRA="-Wl,-Bsymbolic
-Wl,--no-gnu-unique"
CCFLAGS="-fno-gnu-unique"
clang_tidy_toolchain: v4
num_scons_unit_cc_jobs_available: 0.75
compile_variant: *commit-queue
depends_on: []
tasks:
- name: compile_ninja_quick_TG
- name: compile_test_parallel_core_stream_TG
distros:
- amazon2-arm64-xlarge-commitqueue
- name: compile_test_parallel_unittest_stream_TG
distros:
- amazon2-arm64-xlarge-commitqueue
- name: compile_test_parallel_dbtest_stream_TG
distros:
- amazon2-arm64-xlarge-commitqueue
- name: jsCore
distros:
- amazon2-arm64-large
- name: .lint
- name: test_api_version_compatibility
- name: validate_commit_message
- name: lint_large_files_check
- name: check_feature_flag_tags
- name: resmoke_validation_tests
- name: version_gen_validation
distros:
- ubuntu2004-small

View File

@ -69,7 +69,7 @@ modules:
- name: enterprise
repo: git@github.com:10gen/mongo-enterprise-modules.git
prefix: src/mongo/db/modules
branch: master
branch: v7.2
- name: wtdevelop
repo: git@github.com:wiredtiger/wiredtiger.git
@ -676,7 +676,7 @@ functions:
"get buildnumber": &get_buildnumber
command: keyval.inc
params:
key: "${build_variant}_master"
key: "${build_variant}_v7.2"
destination: "builder_num"
"run diskstats": &run_diskstats
@ -2982,15 +2982,7 @@ tasks:
archive-dist-test
task_compile_flags: >-
PREFIX=dist-test
- command: subprocess.exec
params:
binary: bash
add_expansions_to_env: true
args:
- "src/evergreen/run_python_script.sh"
- "evergreen/macos_notary.py"
- "mongodb-binaries.${ext|tgz}"
sign_macos_archive: true
- command: s3.put
params:
optional: true
@ -8095,14 +8087,7 @@ tasks:
${additional_package_targets|}
task_compile_flags: >-
--legacy-tarball
- command: subprocess.exec
params:
binary: bash
add_expansions_to_env: true
args:
- "src/evergreen/run_python_script.sh"
- "evergreen/macos_notary.py"
- "mongodb-dist.${ext|tgz}"
sign_macos_archive: true
- func: "f_expansions_write"
- command: subprocess.exec
params:
@ -9587,6 +9572,13 @@ task_groups:
# - compile_all_but_not_unittests
# - package
- <<: *compile_task_group_template
name: compile_and_package_serial_no_unittests_TG
tasks:
- compile_dist_test
- archive_dist_test
- archive_dist_test_debug
# SERVER-76006
# This is a compile stream meant for non-cached and/or underpowered systems.
# It joins most of the compile tasks together under a single host spread out

View File

@ -33,6 +33,7 @@ variables:
idle_timeout_factor: 1.5
exec_timeout_factor: 1.5
large_distro_name: rhel80-medium
core_analyzer_distro_name: rhel80-large
burn_in_tag_buildvariants: >-
enterprise-rhel-80-64-bit-inmem
enterprise-rhel-80-64-bit-multiversion

View File

@ -25,7 +25,7 @@ variables:
- enterprise
stepback: false
- &linux-x86-compile-params # Essential set of compile parameters used for Linux dev variants.
- &amazon2-x86-compile-params # Essential set of compile parameters used for Linux dev variants.
run_on:
- amazon2
activate: true # These compile variants run on every commit to reduce latency of the auto-reverter.
@ -44,7 +44,7 @@ variables:
--variables-files=etc/scons/mongodbtoolchain_stable_gcc.vars
--link-model=dynamic
- &linux-x86-compile-expansions
- &amazon2-x86-compile-expansions
scons_cache_scope: shared
scons_cache_mode: all
has_packages: false
@ -115,14 +115,14 @@ buildvariants:
distros:
- rhel80-large
- <<: *linux-x86-compile-params
name: &linux-x86-compile linux-x86-compile #static compile
display_name: "Linux x86 Static Library"
- <<: *amazon2-x86-compile-params
name: &amazon2-x86-compile amazon2-x86-compile #static compile
display_name: "Amazon2 x86 Static Compile"
expansions:
<<: *linux-x86-compile-expansions
compile_variant: *linux-x86-compile
<<: *amazon2-x86-compile-expansions
compile_variant: *amazon2-x86-compile
tasks:
- name: compile_test_parallel_core_stream_TG
- name: compile_and_package_serial_no_unittests_TG
- name: generate_buildid_to_debug_symbols_mapping
- name: run_pretty_printer_tests
@ -133,25 +133,10 @@ buildvariants:
<<: *amazon-linux2-arm64-compile-expansions
compile_variant: *amazon-linux2-arm64-compile
tasks:
- name: compile_test_parallel_core_stream_TG
- name: compile_and_package_serial_no_unittests_TG
- name: generate_buildid_to_debug_symbols_mapping
- name: run_pretty_printer_tests
- <<: *generic_linux_compile_params
name: &linux-x86-dynamic-compile-future-tag-multiversion linux-x86-dynamic-compile-future-tag-multiversion
display_name: "Linux x86 Shared Library Compile (future git tag multiversion)"
modules:
- enterprise
expansions:
<<: *linux-x86-dynamic-compile-expansions
bv_future_git_tag: r100.0.0-9999
compile_variant: *linux-x86-dynamic-compile-future-tag-multiversion
depends_on:
- name: version_expansions_gen
variant: enterprise-rhel-80-64-bit-future-git-tag-multiversion-version-gen
tasks:
- name: compile_test_serial_TG
- <<: *generic_linux_compile_params
name: &linux-x86-dynamic-debug-compile-required linux-x86-dynamic-debug-compile-required # TODO: replace with Sanitizer.
display_name: "! Linux x86 Shared Library DEBUG Compile"
@ -173,17 +158,6 @@ buildvariants:
- name: compile_test_parallel_dbtest_stream_TG
- name: generate_buildid_to_debug_symbols_mapping
- <<: *generic_linux_compile_params
name: &linux-x86-dynamic-debug-wtdevelop-compile linux-x86-dynamic-debug-wtdevelop-compile
display_name: "~ Linux WiredTiger develop DEBUG Compile"
activate: false
modules:
- wtdevelop
expansions:
<<: *linux_debug_compile_expansions
use_wt_develop: true
compile_variant: *linux-x86-dynamic-debug-wtdevelop-compile
- <<: *generic_linux_compile_params
name: &linux-debug-aubsan-compile-required linux-debug-aubsan-compile-required
display_name: "! Linux x86 Shared Library {A,UB}SAN Enterprise Compile"

View File

@ -37,6 +37,7 @@ variables:
idle_timeout_factor: 1.5
exec_timeout_factor: 1.5
large_distro_name: rhel80-medium
core_analyzer_distro_name: rhel80-large
buildvariants:
- &enterprise-rhel-80-64-bit-dynamic-config-shard

View File

@ -0,0 +1,10 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>com.apple.security.cs.allow-unsigned-executable-memory</key><true/>
<!--The get-task-allow entitlement is insecure and must only be used in dev environments-->
<key>com.apple.security.get-task-allow</key><true/>
</dict>
</plist>

File diff suppressed because it is too large Load Diff

View File

@ -1,3 +1,4 @@
import argparse
import os
import platform
import shutil
@ -6,15 +7,13 @@ import subprocess
import zipfile
import stat
import sys
import yaml
if platform.system().lower() != 'darwin':
print("Not a macos system, skipping macos signing.")
sys.exit(0)
if len(sys.argv) < 2:
print("Must provide at least 1 archive to sign.")
sys.exit(1)
supported_archs = {
'arm64': 'arm64',
'x86_64': 'amd64'
@ -25,12 +24,20 @@ if arch not in supported_archs:
print(f"Unsupported platform uname arch: {arch}, must be {supported_archs.keys()}")
sys.exit(1)
macnotary_name = f'darwin_{supported_archs[arch]}'
expansions_file = "../expansions.yml"
if not os.path.exists(expansions_file):
print("Evergreen expansions file not found. Skipping macos_notary.")
sys.exit(0)
if os.environ['project'] == "mongodb-mongo-master-nightly":
signing_type = 'notarizeAndSign'
else:
signing_type = 'sign'
with open(expansions_file) as file:
expansions = yaml.safe_load(file)
should_sign = expansions.get("sign_macos_archive", None)
if not should_sign:
print("sign_macos_archive expansion not found not found or false. Skipping macos_notary.")
sys.exit(0)
macnotary_name = f'darwin_{supported_archs[arch]}'
macnotary_url = f'https://macos-notary-1628249594.s3.amazonaws.com/releases/client/latest/{macnotary_name}.zip'
print(f'Fetching macnotary tool from: {macnotary_url}')
@ -42,43 +49,51 @@ st = os.stat(f'{macnotary_name}/macnotary')
os.chmod(f'{macnotary_name}/macnotary', st.st_mode | stat.S_IEXEC)
failed = False
archives = sys.argv[1:]
parser = argparse.ArgumentParser(
prog="MacOS Notary",
description="Sign and/or notarize a tarball containing unsigned binaries.",
)
parser.add_argument("--archive-name", "-a", action="store", required=True)
parser.add_argument("--entitlements-file", "-e", action="store", required=True)
parser.add_argument("--signing-type", "-s", action="store", required=True)
args = parser.parse_args()
archive_name = args.archive_name
entitlements_file = args.entitlements_file
signing_type = args.signing_type
for archive in archives:
archive_base, archive_ext = os.path.splitext(archive)
unsigned_archive = f'{archive_base}_unsigned{archive_ext}'
shutil.move(archive, unsigned_archive)
archive_base, archive_ext = os.path.splitext(archive_name)
unsigned_archive = f'{archive_base}_unsigned{archive_ext}'
shutil.move(archive_name, unsigned_archive)
signing_cmd = [
f'./{macnotary_name}/macnotary',
'-f', f'{unsigned_archive}',
'-m', f'{signing_type}',
'-u', 'https://dev.macos-notary.build.10gen.cc/api',
'-k', 'server',
'--entitlements', 'etc/macos_entitlements.xml',
'--verify',
'-b', 'server.mongodb.com',
'-i', f'{os.environ["task_id"]}',
'-c', f'{os.environ["project"]}',
'-o', f'{archive}'
]
signing_cmd = [
f'./{macnotary_name}/macnotary',
'-f', f'{unsigned_archive}',
'-m', f'{signing_type}',
'-u', 'https://dev.macos-notary.build.10gen.cc/api',
'-k', 'server',
'--entitlements', entitlements_file,
'--verify',
'-b', 'server.mongodb.com',
'-i', f'{expansions["task_id"]}',
'-c', f'{expansions["project"]}',
'-o', f'{archive_name}'
]
signing_env = os.environ.copy()
signing_env['MACOS_NOTARY_SECRET'] = os.environ["macos_notarization_secret"]
print(' '.join(signing_cmd))
p = subprocess.Popen(signing_cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, env=signing_env)
signing_env = os.environ.copy()
signing_env['MACOS_NOTARY_SECRET'] = expansions.get("macos_notarization_secret", "")
print(' '.join(signing_cmd))
p = subprocess.Popen(signing_cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, env=signing_env)
print(f"Signing tool completed with exitcode: {p.returncode}")
for line in iter(p.stdout.readline, b''):
print(f'macnotary: {line.decode("utf-8").strip()}')
p.wait()
print(f"Signing tool completed with exitcode: {p.returncode}")
for line in iter(p.stdout.readline, b''):
print(f'macnotary: {line.decode("utf-8").strip()}')
p.wait()
if p.returncode != 0:
failed = True
shutil.move(unsigned_archive, archive)
else:
os.unlink(unsigned_archive)
if p.returncode != 0:
failed = True
shutil.move(unsigned_archive, archive_name)
else:
os.unlink(unsigned_archive)
if failed:
exit(1)

View File

@ -1,5 +1,9 @@
/**
* Test $_internalShredDocuments parsing and make sure it doesn't modify documents.
* this test assumes {$meta: "indexKey"} will not be missing.
* @tags: [
* do_not_wrap_aggregations_in_facets
* ]
*/
"use strict";
@ -7,7 +11,8 @@ import {assertArrayEq} from "jstests/aggregation/extras/utils.js";
const coll = db[jsTestName()];
coll.insertMany(
[{a: 1, obj: {a: 1}, arr: [{a: 1}]}, {a: 1, obj: {a: 1}, arr: [{a: 1}]}, {}, {a: 1}]);
[{a: 1, obj: {a: 1}, arr: [{a: 1}]}, {a: 2, obj: {a: 1}, arr: [{a: 1}]}, {}, {a: 3}]);
coll.createIndex({a: 1});
assert.commandFailedWithCode(assert.throws(() => coll.aggregate({$_internalShredDocuments: 1})),
7997500);
assert.commandFailedWithCode(
@ -40,3 +45,17 @@ assertNoop(matchExcludeGroup, 0);
assertNoop(matchExcludeGroup, 1);
assertNoop(matchExcludeGroup, 2);
assertNoop(matchExcludeGroup, 3);
assertNoop([{$match: {a: 1}}, {$addFields: {key: {$meta: "indexKey"}}}], 1);
// The shred() function is also used by set windowFields so lets test that too.
const res = coll.aggregate([
{$match: {a: 1}},
{$addFields: {key: {$meta: "indexKey"}}},
{$setWindowFields: {sortBy: {a: 1}, output: {w: {$rank: {}}}}},
{$limit: 1}
])
.toArray();
assert.eq(1, res.length);
assert(res[0].hasOwnProperty("key"));
assert.eq({a: 1}, res[0]["key"]);

View File

@ -1,7 +1,6 @@
/**
* Test that change streams returns shardCollection events with the options specified on the
* original user command.
* TODO SERVER-81138 remove multiversion_incompatible and fix comparison with 7.0 binaries
* @tags: [
* requires_fcv_60,
* requires_sharding,
@ -9,7 +8,6 @@
* change_stream_does_not_expect_txns,
* assumes_unsharded_collection,
* assumes_read_preference_unchanged,
* multiversion_incompatible,
* ]
*/
@ -42,6 +40,19 @@ function getCollectionUuid(coll) {
return collInfo.info.uuid;
}
function assertChangeStreamEventEqMultiversionCompatible(actualEvent, expectedEvent) {
// SERVER-83104: Remove 'numInitialChunks' check once 8.0 becomes last LTS.
if ('numInitialChunks' in actualEvent.operationDescription) {
delete actualEvent.operationDescription.numInitialChunks;
}
// SERVER-83104: Remove 'capped' check once 8.0 becomes last LTS.
if (!('capped' in actualEvent.operationDescription)) {
delete expectedEvent.operationDescription.capped;
}
return assertChangeStreamEventEq(actualEvent, expectedEvent);
}
function assertNextChangeEvent(cursor, expectedEvent) {
let events = test.getNextChanges(cursor, 1);
while (events.length > 0) {
@ -60,7 +71,7 @@ function assertNextChangeEvent(cursor, expectedEvent) {
assert(event.wallTime instanceof Date);
delete event.wallTime;
expectedEvent.collectionUUID = getCollectionUuid(collName);
assertChangeStreamEventEq(event, expectedEvent);
assertChangeStreamEventEqMultiversionCompatible(event, expectedEvent);
}
function runTest(startChangeStream) {
@ -92,7 +103,7 @@ function runTest(startChangeStream) {
}
const shardEvent = events[0];
assertChangeStreamEventEq(shardEvent, expectedOutput);
assertChangeStreamEventEqMultiversionCompatible(shardEvent, expectedOutput);
// Insert a document before starting the next change stream so that we can validate the
// resuming behavior.
@ -181,16 +192,7 @@ function runTest(startChangeStream) {
"unique": false,
"presplitHashedZones": false,
"capped": false,
"collation": {
"locale": "simple",
"caseLevel": false,
"caseFirst": "off",
"strength": 3,
"numericOrdering": false,
"alternate": "non-ignorable",
"maxVariable": "punct",
"normalization": false
}
"collation": {"locale": "simple"}
}
})
}

View File

@ -10,7 +10,6 @@
import {assertDropCollection} from "jstests/libs/collection_drop_recreate.js";
import {FeatureFlagUtil} from "jstests/libs/feature_flag_util.js";
import {checkSBEEnabled} from "jstests/libs/sbe_util.js";
const clustered = db.clusteredColl;
const nonClustered = db.normalColl;
@ -28,6 +27,23 @@ const docs = [{_id: 1, a: 1}, {_id: 2, a: 2}, {_id: 3, a: 3}];
assert.commandWorked(clustered.insertMany(docs));
assert.commandWorked(nonClustered.insertMany(docs));
// When given the recordId of the last record in the collection, we should receive a null
// resumeToken.
const res = assert.commandWorked(db.runCommand({
find: nonClusteredName,
filter: {},
$_requestResumeToken: true,
$_resumeAfter: {'$recordId': NumberLong(3)},
hint: {$natural: 1}
}));
assert(res.hasOwnProperty('cursor'), res);
const cursor = res['cursor'];
assert(cursor.hasOwnProperty('postBatchResumeToken'), res);
// Note that we don't perform an exact equality on 'postBatchResumeToken' because depending on
// the configuration, it may contain additional fields (such as '$initialSyncId').
assert.eq(cursor.postBatchResumeToken.$recordId, null, res);
function validateFailedResumeAfterInFind({collName, resumeAfterSpec, errorCode, explainFail}) {
const spec = {
find: collName,
@ -94,23 +110,12 @@ function testResumeAfter(validateFunction) {
errorCode: ErrorCodes.KeyNotFound
});
if (checkSBEEnabled(db)) {
// This case really means that 'forceClassicEngine' has not been set. It does not mean any
// SBE-specific feature flags are turned on.
validateFunction({
collName: nonClusteredName,
resumeAfterSpec: {'$recordId': null},
errorCode: ErrorCodes.KeyNotFound
});
} else {
assert.commandWorked(db.runCommand({
find: nonClusteredName,
filter: {},
$_requestResumeToken: true,
$_resumeAfter: {'$recordId': null},
hint: {$natural: 1}
}));
}
// Confirm $_resumeAfter token will fail with 'KeyNotFound' if given a null recordId.
validateFunction({
collName: nonClusteredName,
resumeAfterSpec: {'$recordId': null},
errorCode: ErrorCodes.KeyNotFound
});
// Confirm $_resumeAfter will fail to parse if collection does not exist.
validateFunction({

View File

@ -9,6 +9,14 @@
import {getPlanStages, getWinningPlanFromExplain} from "jstests/libs/analyze_plan.js";
const parameterName = "internalQueryEnableBooleanExpressionsSimplifier";
const isSImplifierEnabled =
assert.commandWorked(db.adminCommand({getParameter: 1, [parameterName]: 1}))[parameterName];
if (!isSImplifierEnabled) {
jsTest.log("Skipping the Boolean simplier tests, since the simplifier is disabled...");
quit();
}
/**
* Checks possible representations of an empty filter in query plans, which can be empty object '{}'
* or missing value.

View File

@ -0,0 +1,275 @@
/**
* Test the behavior of $project with pipelines that require the whole document. Specifically, we
* are targeting that rewrites for $project with $getField, or '$$ROOT'/'$$CURRENT' should only
* occur in certain situations.
*
* @tags: [requires_timeseries]
*/
const timeField = "t";
const metaField = "meta1";
const coll = db.timeseries_project_pushdown;
function runTest({docs, pipeline, expectedResults}) {
coll.drop();
assert.commandWorked(db.createCollection(
coll.getName(), {timeseries: {timeField: timeField, metaField: metaField}}));
assert.commandWorked(coll.insertMany(docs));
const results = coll.aggregate(pipeline).toArray();
assert.sameMembers(expectedResults, results, () => {
return `Pipeline: ${tojson(pipeline)}. Explain: ${
tojson(coll.explain().aggregate(pipeline))}`;
});
}
//
// The following tests confirm the behavior of queries in the form: {$getField: "fieldName"}.
//
(function testMeta_OnlyStringField() {
runTest({
docs: [{_id: 1, [timeField]: new Date(), [metaField]: 2}],
pipeline: [{$project: {new: {$getField: metaField}, _id: 0}}],
expectedResults: [{new: 2}]
})
})();
// $getField does not traverse objects, and should not be rewritten when it relies on a mix of
// metaField and measurement fields. Let's validate that behavior.
(function testDottedPath_OnlyStringField() {
runTest({
docs: [
{_id: 1, [timeField]: new Date(), [metaField]: 2, "a.b": 3, a: {b: 2}},
{_id: 2, [timeField]: new Date(), [metaField]: 2, a: {b: 2}} // missing field.
],
pipeline: [{$project: {new: {$add: [`$${metaField}`, {$getField: "a.b"}]}}}],
expectedResults: [{_id: 1, new: 5}, {_id: 2, new: null}]
})
})();
//
// The following tests confirm the behavior of queries in the form: {$getField: {$literal:
// "fieldName"}}.
//
(function testDottedPath_LiteralExpr() {
runTest({
docs: [
{_id: 1, [timeField]: new Date(), [metaField]: 2, "a.$b": 3, a: {b: 4}},
{_id: 2, [timeField]: new Date(), [metaField]: 2, a: {b: 2}} // missing field.
],
pipeline: [{$project: {new: {$add: [`$${metaField}`, {$getField: {$literal: "a.$b"}}]}}}],
expectedResults: [{_id: 1, new: 5}, {_id: 2, new: null}]
})
}());
// There is a difference between the metaField "meta1", and "$meta1". Field paths are allowed to
// have a '$' and are accessed by $getField. We need to validate we return the correct results when
// we have both the metaField and "$meta1".
(function testDollarMeta_LiteralExpr() {
runTest({
docs: [
{_id: 1, [timeField]: new Date(), [metaField]: 2},
{_id: 2, [timeField]: new Date(), [metaField]: 2, "$meta1": 3} // missing field.
],
pipeline: [{$project: {new: {$add: [`$${metaField}`, {$getField: {$literal: "$meta1"}}]}}}],
expectedResults: [{_id: 1, new: null}, {_id: 2, new: 5}]
})
})();
//
// The following tests confirm the behavior of queries in the form: {$getField: <expr>}.
//
(function testMetaField_FieldExpr() {
runTest({
docs: [
{_id: 1, [timeField]: new Date(), [metaField]: 2},
{_id: 2, [timeField]: new Date(), [metaField]: 4, "$meta1": 3}
],
pipeline: [{
$project: {
new: {$add: [`$${metaField}`, {$getField: {$concat: ["m", "e", "t", "a", "1"]}}]}
}
}],
expectedResults: [{_id: 1, new: 4}, {_id: 2, new: 8}]
})
})();
// When we rely on both the metaField and a measurementField we should not perform the rewrite and
// return the correct result.
(function testDottedPath_FieldExpr() {
runTest({
docs: [
{_id: 1, [timeField]: new Date(), [metaField]: 2},
{_id: 2, [timeField]: new Date(), [metaField]: 2, "a.b.c": 3}
],
pipeline: [{
$project:
{new: {$add: [`$${metaField}`, {$getField: {$cond: [false, null, "a.b.c"]}}]}}
}],
expectedResults: [{_id: 1, new: null}, {_id: 2, new: 5}]
})
})();
//
// The following tests confirm the behavior of queries in the form: {$getField: {input: "string",
// field: "string"}}.
//
(function testMetaField_FieldAndInputString() {
runTest({
docs: [
{_id: 1, [timeField]: new Date(), [metaField]: {b: 4}},
{_id: 2, [timeField]: new Date(), [metaField]: {c: 5}} // missing subfield.
],
pipeline: [{$project: {new: {$getField: {input: `$${metaField}`, field: "b"}}}}],
expectedResults: [{_id: 1, new: 4}, {_id: 2}]
})
})();
// Validate the correct results are returned when there is a field with '$' inside the metaField.
// The rewrite should be valid and return the correct field.
(function testMetaFieldObj_FieldAndInputString() {
runTest({
docs: [
{_id: 1, [timeField]: new Date(), [metaField]: {"a.$b": 4}},
{_id: 2, [timeField]: new Date(), [metaField]: {c: 5}} // missing subfield.
],
pipeline: [{$project: {new: {$getField: {input: `$${metaField}`, field: "a.$b"}}}}],
expectedResults: [{_id: 1, new: 4}, {_id: 2}]
})
})();
// When we rely on both the metaField and a measurementField we should not perform the rewrite and
// return the correct result.
(function testDollarMeta_FieldAndInputString() {
runTest({
docs: [
{_id: 1, [timeField]: new Date(), [metaField]: 2, a: {"$meta1": 4}},
{_id: 2, [timeField]: new Date(), [metaField]: 2, a: {c: 5}} // missing subfield.
],
pipeline: [{
$project: {
new: {
$add: [`$${metaField}`, {$getField: {input: "$a", field: {$literal: "$meta1"}}}]
}
}
}],
expectedResults: [{_id: 1, new: 6}, {_id: 2, new: null}]
})
})();
// same test as above but with $addFields and not $project.
(function testDollarMeta_FieldAndInputString_AddFields() {
const time = new Date();
runTest({
docs: [
{_id: 1, [timeField]: time, [metaField]: 2, a: {"$meta1": 4}},
{_id: 2, [timeField]: time, [metaField]: 2, a: {c: 5}} // missing subfield.
],
pipeline: [{
$addFields: {
new: {
$add: [`$${metaField}`, {$getField: {input: "$a", field: {$literal: "$meta1"}}}]
}
}
}],
expectedResults: [
{[timeField]: time, [metaField]: 2, a: {"$meta1": 4}, _id: 1, new: 6},
{[timeField]: time, [metaField]: 2, a: {c: 5}, _id: 2, new: null}
]
})
})();
//
// The following tests confirm the behavior of queries in the form: {$getField: {input: <expr>,
// field: <expr>}}.
//
(function testMetaField_FieldAndInputExpr() {
runTest({
docs: [
{_id: 1, [timeField]: new Date(), [metaField]: {field: 3}},
{_id: 2, [timeField]: new Date(), [metaField]: {notField: 4}} // missing subfield.
],
pipeline: [{
$project: {
new: {
$getField: {
input: {$cond: [true, "$meta1", null]},
field: {$concat: ["f", "i", "e", "l", "d"]}
}
}
}
}],
expectedResults: [{_id: 1, new: 3}, {_id: 2}]
})
})();
// When we rely on both the metaField and a measurementField we should not perform the rewrite and
// return the correct result.
(function testDottedPath_FieldAndInputExpr() {
runTest({
docs: [
{_id: 1, [timeField]: new Date(), [metaField]: 2, a: {"b.c": 3}},
{_id: 2, [timeField]: new Date(), [metaField]: 2, a: {b: {c: 4}}} // missing field.
],
pipeline: [{
$project: {
new: {
$add: [
`$${metaField}`,
{
$getField: {
input: {$cond: [true, "$a", null]},
field: {$concat: ["b", ".", "c"]}
}
}
]
}
}
}],
expectedResults: [{_id: 1, new: 5}, {_id: 2, new: null}]
})
})();
// This test validates that $project with '$$ROOT' which requires the whole document returns the
// correct results.
(function testProject_WithROOT() {
const time = new Date();
runTest({
docs: [
{_id: 1, [timeField]: time, [metaField]: 2, a: 2},
{_id: 2, [timeField]: time, [metaField]: 2, b: 3}
],
pipeline: [{$project: {new: "$$ROOT", _id: 1}}],
expectedResults: [
{_id: 1, new: {_id: 1, [timeField]: time, [metaField]: 2, a: 2}},
{_id: 2, new: {_id: 2, [timeField]: time, [metaField]: 2, b: 3}}
]
})
})();
(function testAddFields_WithROOT() {
const time = new Date();
runTest({
docs: [
{_id: 1, [timeField]: time, [metaField]: 2, a: 2},
{_id: 2, [timeField]: time, [metaField]: 2, b: 3}
],
pipeline: [{$addFields: {new: "$$ROOT"}}],
expectedResults: [
{
_id: 1,
[timeField]: time,
[metaField]: 2,
a: 2,
new: {_id: 1, [timeField]: time, [metaField]: 2, a: 2}
},
{
_id: 2,
[timeField]: time,
[metaField]: 2,
b: 3,
new: {_id: 2, [timeField]: time, [metaField]: 2, b: 3}
}
]
})
})();

View File

@ -16,7 +16,6 @@
* ]
*/
import {TimeseriesTest} from "jstests/core/timeseries/libs/timeseries.js";
import {checkSBEEnabled} from "jstests/libs/sbe_util.js";
TimeseriesTest.run((insert) => {
const timeFieldName = "time";
@ -79,36 +78,18 @@ TimeseriesTest.run((insert) => {
assert.hasFields(res.cursor, ["postBatchResumeToken"]);
resumeToken = res.cursor.postBatchResumeToken;
// TODO SERVER-79848: the behavior of the two engines should match, but it doesn't. We might
// also get null record id in the resume token during FCV upgrade/downgrade.
if (!checkSBEEnabled(db, ["featureFlagTimeSeriesInSbe"])) {
// In the classic engine, after a collection is exhausted, the record id in the resume token
// is set to null.
assert.eq(null, resumeToken.$recordId, "Got resume token " + tojson(resumeToken));
// After a collection is exhausted, the record id in the resume token is set to null.
assert.eq(null, resumeToken.$recordId, "Got resume token " + tojson(resumeToken));
// Try to resume from a null '$recordId'. It should reset to the start of the collection.
res = assert.commandWorked(db.runCommand({
find: bucketsColl.getName(),
hint: {$natural: 1},
batchSize: 1,
$_requestResumeToken: true,
$_resumeAfter: resumeToken
}));
assert.neq([], res.cursor.firstBatch, "Expect some data to be returned");
assert.hasFields(res.cursor, ["postBatchResumeToken"]);
resumeToken = res.cursor.postBatchResumeToken;
assert.neq(null, resumeToken.$recordId, "Got resume token " + tojson(resumeToken));
} else {
// In SBE we might get the last token back or we might get null. Don't check _what_ we get,
// just check that we can use it.
assert.commandWorked(db.runCommand({
find: bucketsColl.getName(),
hint: {$natural: 1},
batchSize: 1,
$_requestResumeToken: true,
$_resumeAfter: resumeToken
}));
}
// Try to resume from a null '$recordId'. It should fail with a 'KeyNotFound' error.
assert.commandFailedWithCode(db.runCommand({
find: bucketsColl.getName(),
hint: {$natural: 1},
batchSize: 1,
$_requestResumeToken: true,
$_resumeAfter: resumeToken
}),
ErrorCodes.KeyNotFound);
// Test that '$_resumeAfter' fails if the recordId is Long.
assert.commandFailedWithCode(db.runCommand({

View File

@ -0,0 +1,72 @@
/**
* Tests that the size storer entry for a collection gets cleaned up when that collection is
* dropped.
*
* @tags: [
* incompatible_with_macos, # TODO (SERVER-83337): Re-enable on macOS.
* requires_replication,
* requires_wiredtiger,
* ]
*/
import {
runWiredTigerTool,
} from "jstests/disk/libs/wt_file_helper.js";
const replTest = new ReplSetTest({nodes: 1});
replTest.startSet();
replTest.initiate();
let primary = replTest.getPrimary();
const dbpath = primary.dbpath;
const coll = function() {
return primary.getDB(jsTestName()).test;
};
// TODO (SERVER-82902): Use JSON-formatted size storer data.
// const getSizeStorerData = function() {
// const filePath = dbpath + (_isWindows() ? "\\" : "/") + jsTestName();
// runWiredTigerTool("-r", "-h", dbpath, "dump", "-j", "-f", filePath, "sizeStorer");
// return JSON.parse(cat(filePath))["table:sizeStorer"][1].data;
// };
const getSizeStorerData = function() {
const filePath = dbpath + (_isWindows() ? "\\" : "/") + jsTestName();
runWiredTigerTool("-r", "-h", dbpath, "dump", "-f", filePath, "sizeStorer");
return cat(filePath);
};
assert.commandWorked(coll().insert({a: 1}));
assert.eq(coll().count(), 1);
const uri = coll().stats().wiredTiger.uri.split("statistics:")[1];
replTest.stop(primary, undefined, {}, {forRestart: true});
let sizeStorerData = getSizeStorerData();
// TODO (SERVER-82902): Use JSON-formatted size storer data.
// assert(sizeStorerData.find(entry => entry.key0 === uri),
// "Size storer unexpectedly does not contain entry for " + uri + ": " +
// tojson(sizeStorerData));
assert(sizeStorerData.includes(uri),
"Size storer unexpectedly does not contain entry for " + uri + ": " + sizeStorerData);
replTest.start(
primary, {setParameter: {minSnapshotHistoryWindowInSeconds: 0}}, true /* forRestart */);
primary = replTest.getPrimary();
assert.eq(coll().count(), 1);
assert(coll().drop());
assert.commandWorked(primary.adminCommand({appendOplogNote: 1, data: {msg: "advance timestamp"}}));
assert.commandWorked(primary.adminCommand({fsync: 1}));
replTest.stop(primary, undefined, {}, {forRestart: true});
sizeStorerData = getSizeStorerData();
// TODO (SERVER-82902): Use JSON-formatted size storer data.
// assert(!sizeStorerData.find(entry => entry.key0 === uri),
// "Size storer unexpectedly contains entry for " + uri + ": " + tojson(sizeStorerData));
assert(!sizeStorerData.includes(uri),
"Size storer unexpectedly contains entry for " + uri + ": " + sizeStorerData);
replTest.start(primary, {}, true /* forRestart */);
replTest.stopSet();

View File

@ -0,0 +1,69 @@
/**
* Tests that the size storer entry for a collection gets cleaned up when that collection is
* dropped.
*
* @tags: [
* requires_wiredtiger,
* ]
*/
import {
runWiredTigerTool,
} from "jstests/disk/libs/wt_file_helper.js";
const runTest = function(insertAfterRestart) {
let conn = MongoRunner.runMongod();
const dbpath = conn.dbpath;
const coll = function() {
return conn.getDB(jsTestName()).test;
};
// TODO (SERVER-82902): Use JSON-formatted size storer data.
// const getSizeStorerData = function() {
// const filePath = dbpath + (_isWindows() ? "\\" : "/") + jsTestName();
// runWiredTigerTool("-r", "-h", dbpath, "dump", "-j", "-f", filePath, "sizeStorer");
// return JSON.parse(cat(filePath))["table:sizeStorer"][1].data;
// };
const getSizeStorerData = function() {
const filePath = dbpath + (_isWindows() ? "\\" : "/") + jsTestName();
runWiredTigerTool("-r", "-h", dbpath, "dump", "-f", filePath, "sizeStorer");
return cat(filePath);
};
assert.commandWorked(coll().insert({a: 1}));
assert.eq(coll().count(), 1);
const uri = coll().stats().wiredTiger.uri.split("statistics:")[1];
MongoRunner.stopMongod(conn);
let sizeStorerData = getSizeStorerData();
// TODO (SERVER-82902): Use JSON-formatted size storer data.
// assert(sizeStorerData.find(entry => entry.key0 === uri),
// "Size storer unexpectedly does not contain entry for " + uri + ": " +
// tojson(sizeStorerData));
assert(sizeStorerData.includes(uri),
"Size storer unexpectedly does not contain entry for " + uri + ": " + sizeStorerData);
conn = MongoRunner.runMongod({dbpath: dbpath, noCleanData: true, setParameter: {syncdelay: 0}});
if (insertAfterRestart) {
assert.commandWorked(coll().insert({a: 2}));
}
assert.eq(coll().count(), insertAfterRestart ? 2 : 1);
assert(coll().drop());
assert.commandWorked(conn.adminCommand({setParameter: 1, syncdelay: 1}));
checkLog.containsJson(conn, 6776600, {ident: uri.split("table:")[1]});
MongoRunner.stopMongod(conn);
sizeStorerData = getSizeStorerData();
// TODO (SERVER-82902): Use JSON-formatted size storer data.
// assert(!sizeStorerData.find(entry => entry.key0 === uri),
// "Size storer unexpectedly contains entry for " + uri + ": " + tojson(sizeStorerData));
assert(!sizeStorerData.includes(uri),
"Size storer unexpectedly contains entry for " + uri + ": " + sizeStorerData);
};
runTest(false);
runTest(true);

View File

@ -60,7 +60,7 @@ const sendFCVUpDown = function(ver) {
};
Random.setRandomSeed();
let maxSleep = 5000; // 5 sec.
let maxSleep = 1000; // 1 sec.
let currSleep = 10; // Start at 10ms.
// Get time interval to sleep in ms.

View File

@ -207,7 +207,7 @@ export function assertExpectedResults(results,
expectedDocsReturnedMin,
expectedDocsReturnedSumOfSq,
getMores) {
const {key, metrics} = results;
const {key, metrics, asOf} = results;
confirmAllExpectedFieldsPresent(expectedQueryStatsKey, key);
assert.eq(expectedExecCount, metrics.execCount);
assert.docEq({
@ -231,6 +231,7 @@ export function assertExpectedResults(results,
assert.neq(lastExecutionMicros, NumberLong(0));
assert.neq(firstSeenTimestamp.getTime(), 0);
assert.neq(latestSeenTimestamp.getTime(), 0);
assert.neq(asOf.getTime(), 0);
const distributionFields = ['sum', 'max', 'min', 'sumOfSquares'];
for (const field of distributionFields) {

View File

@ -0,0 +1,53 @@
/**
* Tests that dbCheck ignores prepared updates on secondaries.
*/
import {PrepareHelpers} from "jstests/core/txns/libs/prepare_helpers.js";
import {configureFailPoint} from "jstests/libs/fail_point_util.js";
import {awaitDbCheckCompletion, checkHealthLog} from "jstests/replsets/libs/dbcheck_utils.js";
const replSet = new ReplSetTest({nodes: 2});
replSet.startSet();
replSet.initiate();
const primary = replSet.getPrimary();
const secondary = replSet.getSecondary();
const dbName = 'test';
const db = primary.getDB(dbName);
const collName = 'coll';
const coll = db[collName];
assert.commandWorked(coll.insert({_id: 0, a: "first"}));
// Stop dbCheck after it scans but before it writes the oplog entry.
const failPoint = configureFailPoint(primary, "hangBeforeDbCheckLogOp");
const awaitDbCheck = startParallelShell(() => {
assert.commandWorked(db.runCommand({dbCheck: 'coll'}));
}, primary.port);
failPoint.wait();
// Insert a document in a prepared state in the range that dbCheck has just validated.
const session = primary.startSession();
const sessionDB = session.getDatabase(dbName);
const sessionColl = sessionDB[collName];
session.startTransaction();
assert.commandWorked(sessionColl.insert({_id: 1}));
PrepareHelpers.prepareTransaction(session);
// Let dbCheck replicate to the secondary and ensure it does not hit a prepare conflict.
failPoint.off();
awaitDbCheck();
awaitDbCheckCompletion(replSet, db);
// We should not find inconsistencies on any node.
[primary, secondary].forEach((node) => {
checkHealthLog(node.getDB('local').system.healthlog, {severity: "error"}, 0);
});
session.abortTransaction();
replSet.stopSet(undefined /* signal */, false /* forRestart */);

View File

@ -5,7 +5,6 @@
* @tags: [
* does_not_support_stepdowns,
* requires_fcv_70,
* multiversion_incompatible,
* ]
*/
import {DiscoverTopology} from "jstests/libs/discover_topology.js";
@ -45,6 +44,16 @@ function verifyOpEntriesOnNodes(expectedOpEntryTemplates, nodes) {
assert.eq(expectedOpEntryTemplates.length, foundOpEntries.length);
for (let i = 0; i < foundOpEntries.length; ++i) {
// SERVER-83104: Remove 'numInitialChunks' check once 8.0 becomes last LTS.
if ('numInitialChunks' in foundOpEntries[i].o2) {
delete foundOpEntries[i].o2.numInitialChunks;
}
// SERVER-83104: Remove 'capped' check once 8.0 becomes last LTS.
if (!('capped' in foundOpEntries[i].o2)) {
delete expectedOpEntryTemplates[i].o2.capped;
}
assert.eq(expectedOpEntryTemplates[i].op, foundOpEntries[i].op);
assert.eq(expectedOpEntryTemplates[i].ns, foundOpEntries[i].ns);
assert.docEq(expectedOpEntryTemplates[i].o, foundOpEntries[i].o);

View File

@ -1,7 +1,7 @@
/**
* Tests that direct shard connections are correctly allowed and disallowed using authentication.
*
* @tags: [featureFlagCheckForDirectShardOperations, requires_fcv_71]
* @tags: [featureFlagCheckForDirectShardOperations, requires_fcv_72]
*/
// Create a new sharded cluster for testing and enable auth.
const st = new ShardingTest({name: jsTestName(), keyFile: "jstests/libs/key1", shards: 1});
@ -87,4 +87,4 @@ jsTest.log("Running tests with two shards.");
// Stop the sharding test before the additional shard to ensure the test hooks run successfully.
st.stop();
newShard.stopSet();
newShard.stopSet();

View File

@ -2,7 +2,12 @@
* Testing that config.tags are correctly updated after resharding hashed shard key with zones.
*/
const st = new ShardingTest({shard: 2});
const st = new ShardingTest({
shard: 2,
configOptions:
{setParameter:
{'reshardingCriticalSectionTimeoutMillis': 24 * 60 * 60 * 1000 /* 1 day */}}
});
const dbName = "testDb";
const collName = "testColl";
const ns = dbName + "." + collName;

View File

@ -1,6 +1,13 @@
import {FeatureFlagUtil} from "jstests/libs/feature_flag_util.js";
const st = new ShardingTest({shards: 3, chunkSize: 1});
const st = new ShardingTest({
shards: 3,
chunkSize: 1,
configOptions:
{setParameter:
{reshardingCriticalSectionTimeoutMillis: 24 * 60 * 60 * 1000, /* 1 day */}}
});
const configDB = st.s.getDB('config');
const shard0 = st.shard0.shardName;
const shard1 = st.shard1.shardName;

102
poetry.lock generated
View File

@ -1133,72 +1133,6 @@ files = [
docs = ["Sphinx"]
test = ["objgraph", "psutil"]
[[package]]
name = "grpcio"
version = "1.59.0"
description = "HTTP/2-based RPC framework"
optional = false
python-versions = ">=3.7"
files = [
{file = "grpcio-1.59.0-cp310-cp310-linux_armv7l.whl", hash = "sha256:225e5fa61c35eeaebb4e7491cd2d768cd8eb6ed00f2664fa83a58f29418b39fd"},
{file = "grpcio-1.59.0-cp310-cp310-macosx_12_0_universal2.whl", hash = "sha256:b95ec8ecc4f703f5caaa8d96e93e40c7f589bad299a2617bdb8becbcce525539"},
{file = "grpcio-1.59.0-cp310-cp310-manylinux_2_17_aarch64.whl", hash = "sha256:1a839ba86764cc48226f50b924216000c79779c563a301586a107bda9cbe9dcf"},
{file = "grpcio-1.59.0-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:f6cfe44a5d7c7d5f1017a7da1c8160304091ca5dc64a0f85bca0d63008c3137a"},
{file = "grpcio-1.59.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:d0fcf53df684fcc0154b1e61f6b4a8c4cf5f49d98a63511e3f30966feff39cd0"},
{file = "grpcio-1.59.0-cp310-cp310-musllinux_1_1_i686.whl", hash = "sha256:fa66cac32861500f280bb60fe7d5b3e22d68c51e18e65367e38f8669b78cea3b"},
{file = "grpcio-1.59.0-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:8cd2d38c2d52f607d75a74143113174c36d8a416d9472415eab834f837580cf7"},
{file = "grpcio-1.59.0-cp310-cp310-win32.whl", hash = "sha256:228b91ce454876d7eed74041aff24a8f04c0306b7250a2da99d35dd25e2a1211"},
{file = "grpcio-1.59.0-cp310-cp310-win_amd64.whl", hash = "sha256:ca87ee6183421b7cea3544190061f6c1c3dfc959e0b57a5286b108511fd34ff4"},
{file = "grpcio-1.59.0-cp311-cp311-linux_armv7l.whl", hash = "sha256:c173a87d622ea074ce79be33b952f0b424fa92182063c3bda8625c11d3585d09"},
{file = "grpcio-1.59.0-cp311-cp311-macosx_10_10_universal2.whl", hash = "sha256:ec78aebb9b6771d6a1de7b6ca2f779a2f6113b9108d486e904bde323d51f5589"},
{file = "grpcio-1.59.0-cp311-cp311-manylinux_2_17_aarch64.whl", hash = "sha256:0b84445fa94d59e6806c10266b977f92fa997db3585f125d6b751af02ff8b9fe"},
{file = "grpcio-1.59.0-cp311-cp311-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:c251d22de8f9f5cca9ee47e4bade7c5c853e6e40743f47f5cc02288ee7a87252"},
{file = "grpcio-1.59.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:956f0b7cb465a65de1bd90d5a7475b4dc55089b25042fe0f6c870707e9aabb1d"},
{file = "grpcio-1.59.0-cp311-cp311-musllinux_1_1_i686.whl", hash = "sha256:38da5310ef84e16d638ad89550b5b9424df508fd5c7b968b90eb9629ca9be4b9"},
{file = "grpcio-1.59.0-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:63982150a7d598281fa1d7ffead6096e543ff8be189d3235dd2b5604f2c553e5"},
{file = "grpcio-1.59.0-cp311-cp311-win32.whl", hash = "sha256:50eff97397e29eeee5df106ea1afce3ee134d567aa2c8e04fabab05c79d791a7"},
{file = "grpcio-1.59.0-cp311-cp311-win_amd64.whl", hash = "sha256:15f03bd714f987d48ae57fe092cf81960ae36da4e520e729392a59a75cda4f29"},
{file = "grpcio-1.59.0-cp312-cp312-linux_armv7l.whl", hash = "sha256:f1feb034321ae2f718172d86b8276c03599846dc7bb1792ae370af02718f91c5"},
{file = "grpcio-1.59.0-cp312-cp312-macosx_10_10_universal2.whl", hash = "sha256:d09bd2a4e9f5a44d36bb8684f284835c14d30c22d8ec92ce796655af12163588"},
{file = "grpcio-1.59.0-cp312-cp312-manylinux_2_17_aarch64.whl", hash = "sha256:2f120d27051e4c59db2f267b71b833796770d3ea36ca712befa8c5fff5da6ebd"},
{file = "grpcio-1.59.0-cp312-cp312-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:ba0ca727a173ee093f49ead932c051af463258b4b493b956a2c099696f38aa66"},
{file = "grpcio-1.59.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:5711c51e204dc52065f4a3327dca46e69636a0b76d3e98c2c28c4ccef9b04c52"},
{file = "grpcio-1.59.0-cp312-cp312-musllinux_1_1_i686.whl", hash = "sha256:d74f7d2d7c242a6af9d4d069552ec3669965b74fed6b92946e0e13b4168374f9"},
{file = "grpcio-1.59.0-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:3859917de234a0a2a52132489c4425a73669de9c458b01c9a83687f1f31b5b10"},
{file = "grpcio-1.59.0-cp312-cp312-win32.whl", hash = "sha256:de2599985b7c1b4ce7526e15c969d66b93687571aa008ca749d6235d056b7205"},
{file = "grpcio-1.59.0-cp312-cp312-win_amd64.whl", hash = "sha256:598f3530231cf10ae03f4ab92d48c3be1fee0c52213a1d5958df1a90957e6a88"},
{file = "grpcio-1.59.0-cp37-cp37m-linux_armv7l.whl", hash = "sha256:b34c7a4c31841a2ea27246a05eed8a80c319bfc0d3e644412ec9ce437105ff6c"},
{file = "grpcio-1.59.0-cp37-cp37m-macosx_10_10_universal2.whl", hash = "sha256:c4dfdb49f4997dc664f30116af2d34751b91aa031f8c8ee251ce4dcfc11277b0"},
{file = "grpcio-1.59.0-cp37-cp37m-manylinux_2_17_aarch64.whl", hash = "sha256:61bc72a00ecc2b79d9695220b4d02e8ba53b702b42411397e831c9b0589f08a3"},
{file = "grpcio-1.59.0-cp37-cp37m-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:f367e4b524cb319e50acbdea57bb63c3b717c5d561974ace0b065a648bb3bad3"},
{file = "grpcio-1.59.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:849c47ef42424c86af069a9c5e691a765e304079755d5c29eff511263fad9c2a"},
{file = "grpcio-1.59.0-cp37-cp37m-musllinux_1_1_i686.whl", hash = "sha256:c0488c2b0528e6072010182075615620071371701733c63ab5be49140ed8f7f0"},
{file = "grpcio-1.59.0-cp37-cp37m-musllinux_1_1_x86_64.whl", hash = "sha256:611d9aa0017fa386809bddcb76653a5ab18c264faf4d9ff35cb904d44745f575"},
{file = "grpcio-1.59.0-cp37-cp37m-win_amd64.whl", hash = "sha256:e5378785dce2b91eb2e5b857ec7602305a3b5cf78311767146464bfa365fc897"},
{file = "grpcio-1.59.0-cp38-cp38-linux_armv7l.whl", hash = "sha256:fe976910de34d21057bcb53b2c5e667843588b48bf11339da2a75f5c4c5b4055"},
{file = "grpcio-1.59.0-cp38-cp38-macosx_10_10_universal2.whl", hash = "sha256:c041a91712bf23b2a910f61e16565a05869e505dc5a5c025d429ca6de5de842c"},
{file = "grpcio-1.59.0-cp38-cp38-manylinux_2_17_aarch64.whl", hash = "sha256:0ae444221b2c16d8211b55326f8ba173ba8f8c76349bfc1768198ba592b58f74"},
{file = "grpcio-1.59.0-cp38-cp38-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:ceb1e68135788c3fce2211de86a7597591f0b9a0d2bb80e8401fd1d915991bac"},
{file = "grpcio-1.59.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:6c4b1cc3a9dc1924d2eb26eec8792fedd4b3fcd10111e26c1d551f2e4eda79ce"},
{file = "grpcio-1.59.0-cp38-cp38-musllinux_1_1_i686.whl", hash = "sha256:871371ce0c0055d3db2a86fdebd1e1d647cf21a8912acc30052660297a5a6901"},
{file = "grpcio-1.59.0-cp38-cp38-musllinux_1_1_x86_64.whl", hash = "sha256:93e9cb546e610829e462147ce724a9cb108e61647a3454500438a6deef610be1"},
{file = "grpcio-1.59.0-cp38-cp38-win32.whl", hash = "sha256:f21917aa50b40842b51aff2de6ebf9e2f6af3fe0971c31960ad6a3a2b24988f4"},
{file = "grpcio-1.59.0-cp38-cp38-win_amd64.whl", hash = "sha256:14890da86a0c0e9dc1ea8e90101d7a3e0e7b1e71f4487fab36e2bfd2ecadd13c"},
{file = "grpcio-1.59.0-cp39-cp39-linux_armv7l.whl", hash = "sha256:34341d9e81a4b669a5f5dca3b2a760b6798e95cdda2b173e65d29d0b16692857"},
{file = "grpcio-1.59.0-cp39-cp39-macosx_10_10_universal2.whl", hash = "sha256:986de4aa75646e963466b386a8c5055c8b23a26a36a6c99052385d6fe8aaf180"},
{file = "grpcio-1.59.0-cp39-cp39-manylinux_2_17_aarch64.whl", hash = "sha256:aca8a24fef80bef73f83eb8153f5f5a0134d9539b4c436a716256b311dda90a6"},
{file = "grpcio-1.59.0-cp39-cp39-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:936b2e04663660c600d5173bc2cc84e15adbad9c8f71946eb833b0afc205b996"},
{file = "grpcio-1.59.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:fc8bf2e7bc725e76c0c11e474634a08c8f24bcf7426c0c6d60c8f9c6e70e4d4a"},
{file = "grpcio-1.59.0-cp39-cp39-musllinux_1_1_i686.whl", hash = "sha256:81d86a096ccd24a57fa5772a544c9e566218bc4de49e8c909882dae9d73392df"},
{file = "grpcio-1.59.0-cp39-cp39-musllinux_1_1_x86_64.whl", hash = "sha256:2ea95cd6abbe20138b8df965b4a8674ec312aaef3147c0f46a0bac661f09e8d0"},
{file = "grpcio-1.59.0-cp39-cp39-win32.whl", hash = "sha256:3b8ff795d35a93d1df6531f31c1502673d1cebeeba93d0f9bd74617381507e3f"},
{file = "grpcio-1.59.0-cp39-cp39-win_amd64.whl", hash = "sha256:38823bd088c69f59966f594d087d3a929d1ef310506bee9e3648317660d65b81"},
{file = "grpcio-1.59.0.tar.gz", hash = "sha256:acf70a63cf09dd494000007b798aff88a436e1c03b394995ce450be437b8e54f"},
]
[package.extras]
protobuf = ["grpcio-tools (>=1.59.0)"]
[[package]]
name = "h11"
version = "0.14.0"
@ -1694,6 +1628,16 @@ files = [
{file = "MarkupSafe-2.1.3-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:5bbe06f8eeafd38e5d0a4894ffec89378b6c6a625ff57e3028921f8ff59318ac"},
{file = "MarkupSafe-2.1.3-cp311-cp311-win32.whl", hash = "sha256:dd15ff04ffd7e05ffcb7fe79f1b98041b8ea30ae9234aed2a9168b5797c3effb"},
{file = "MarkupSafe-2.1.3-cp311-cp311-win_amd64.whl", hash = "sha256:134da1eca9ec0ae528110ccc9e48041e0828d79f24121a1a146161103c76e686"},
{file = "MarkupSafe-2.1.3-cp312-cp312-macosx_10_9_universal2.whl", hash = "sha256:f698de3fd0c4e6972b92290a45bd9b1536bffe8c6759c62471efaa8acb4c37bc"},
{file = "MarkupSafe-2.1.3-cp312-cp312-macosx_10_9_x86_64.whl", hash = "sha256:aa57bd9cf8ae831a362185ee444e15a93ecb2e344c8e52e4d721ea3ab6ef1823"},
{file = "MarkupSafe-2.1.3-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ffcc3f7c66b5f5b7931a5aa68fc9cecc51e685ef90282f4a82f0f5e9b704ad11"},
{file = "MarkupSafe-2.1.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:47d4f1c5f80fc62fdd7777d0d40a2e9dda0a05883ab11374334f6c4de38adffd"},
{file = "MarkupSafe-2.1.3-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:1f67c7038d560d92149c060157d623c542173016c4babc0c1913cca0564b9939"},
{file = "MarkupSafe-2.1.3-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:9aad3c1755095ce347e26488214ef77e0485a3c34a50c5a5e2471dff60b9dd9c"},
{file = "MarkupSafe-2.1.3-cp312-cp312-musllinux_1_1_i686.whl", hash = "sha256:14ff806850827afd6b07a5f32bd917fb7f45b046ba40c57abdb636674a8b559c"},
{file = "MarkupSafe-2.1.3-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:8f9293864fe09b8149f0cc42ce56e3f0e54de883a9de90cd427f191c346eb2e1"},
{file = "MarkupSafe-2.1.3-cp312-cp312-win32.whl", hash = "sha256:715d3562f79d540f251b99ebd6d8baa547118974341db04f5ad06d5ea3eb8007"},
{file = "MarkupSafe-2.1.3-cp312-cp312-win_amd64.whl", hash = "sha256:1b8dd8c3fd14349433c79fa8abeb573a55fc0fdd769133baac1f5e07abf54aeb"},
{file = "MarkupSafe-2.1.3-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:8e254ae696c88d98da6555f5ace2279cf7cd5b3f52be2b5cf97feafe883b58d2"},
{file = "MarkupSafe-2.1.3-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:cb0932dc158471523c9637e807d9bfb93e06a95cbf010f1a38b98623b929ef2b"},
{file = "MarkupSafe-2.1.3-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:9402b03f1a1b4dc4c19845e5c749e3ab82d5078d16a2a4c2cd2df62d57bb0707"},
@ -2071,30 +2015,6 @@ files = [
backoff = {version = ">=1.10.0,<3.0.0", markers = "python_version >= \"3.7\""}
opentelemetry-proto = "1.20.0"
[[package]]
name = "opentelemetry-exporter-otlp-proto-grpc"
version = "1.20.0"
description = "OpenTelemetry Collector Protobuf over gRPC Exporter"
optional = false
python-versions = ">=3.7"
files = [
{file = "opentelemetry_exporter_otlp_proto_grpc-1.20.0-py3-none-any.whl", hash = "sha256:7c3f066065891b56348ba2c7f9df6ec635a712841cae0a36f2f6a81642ae7dec"},
{file = "opentelemetry_exporter_otlp_proto_grpc-1.20.0.tar.gz", hash = "sha256:6c06d43c3771bda1795226e327722b4b980fa1ca1ec9e985f2ef3e29795bdd52"},
]
[package.dependencies]
backoff = {version = ">=1.10.0,<3.0.0", markers = "python_version >= \"3.7\""}
deprecated = ">=1.2.6"
googleapis-common-protos = ">=1.52,<2.0"
grpcio = ">=1.0.0,<2.0.0"
opentelemetry-api = ">=1.15,<2.0"
opentelemetry-exporter-otlp-proto-common = "1.20.0"
opentelemetry-proto = "1.20.0"
opentelemetry-sdk = ">=1.20.0,<1.21.0"
[package.extras]
test = ["pytest-grpc"]
[[package]]
name = "opentelemetry-proto"
version = "1.20.0"
@ -4344,4 +4264,4 @@ libdeps = ["cxxfilt", "eventlet", "flask", "flask-cors", "gevent", "lxml", "prog
[metadata]
lock-version = "2.0"
python-versions = ">=3.9,<4.0"
content-hash = "a71db329bfe251a27ae909e027c9eceb8fd073ac088e5ee20aa0171a8b5f3ba1"
content-hash = "e57800fd001dcce6b3544f859ff64a92de221217890bfa001cf887d942000432"

View File

@ -149,16 +149,10 @@ selenium = "^4.9.1"
geckodriver-autoinstaller = "^0.1.0"
retry = "^0.9.2"
gdbmongo = "^0.14.0"
googleapis-common-protos = "^1.61.0"
opentelemetry-api = "*"
opentelemetry-sdk = "*"
# TODO: EVG-20576
# Once the above is merged we can get rid of our dependency on grpc
# TODO: SERVER-80336 grpc has problems on macosx, ppc, and s390x.
# For now lets just disable otel metrics on these variants
opentelemetry-exporter-otlp-proto-grpc = [
{ version = "*", markers = "platform_machine != 'darwin' and platform_machine != 's390x' and platform_machine != 'ppc64le'" },
]
opentelemetry-exporter-otlp-proto-common = "*"
[tool.poetry.group.tooling-metrics.dependencies]
mongo-tooling-metrics = "1.0.8"

View File

@ -290,6 +290,25 @@ def archive_builder(source, target, env, for_signature):
return cmd
def macos_archive_sign_builder(source, target, env, for_signature):
if env['PLATFORM'] != 'darwin' or env.GetOption("ninja") != 'disabled':
return ""
if env.GetOption("release") is not None:
print("MacOS release build found, signing with release entitlements.")
entitlements_file = 'etc/macos_release_entitlements.xml'
signing_type = 'notarizeAndSign'
else:
print("MacOS dev build found, signing with insecure development entitlements.")
entitlements_file = 'etc/macos_dev_entitlements.xml'
signing_type = 'sign'
archive_name = env.File(target[0])
macos_notory_cmd = f"{sys.executable} evergreen/macos_notary.py --archive-name={archive_name} --entitlements-file={entitlements_file} --signing-type={signing_type}"
return macos_notory_cmd
def exists(env):
return True
@ -299,11 +318,14 @@ def generate(env):
env.Tool("auto_install_binaries")
bld = SCons.Builder.Builder(
action=SCons.Action.CommandGeneratorAction(
archive_builder,
{"cmdstr": "Building package ${TARGETS[0]} from ${SOURCES[1:]}"}
if not env.Verbose() else {"cmdstr": ""},
))
action=SCons.Action.ListAction([
SCons.Action.CommandGeneratorAction(
archive_builder,
{"cmdstr": "Building package ${TARGETS[0]} from ${SOURCES[1:]}"}
if not env.Verbose() else {"cmdstr": ""},
),
SCons.Action.CommandGeneratorAction(macos_archive_sign_builder, {})
]))
env.Append(BUILDERS={"AutoArchive": bld})
env["AUTO_ARCHIVE_TARBALL_SUFFIX"] = env.get(
"AUTO_ARCHIVE_TARBALL_SUFFIX",

View File

@ -120,15 +120,17 @@ std::pair<std::string, std::string> doDeviceAuthorizationGrantFlow(
deviceAuthorizationEndpoint.startsWith("http://localhost"_sd));
auto clientId = serverReply.getClientId();
uassert(ErrorCodes::BadValue, "Encountered empty client ID in server reply", !clientId.empty());
uassert(ErrorCodes::BadValue,
"Encountered empty client ID in server reply",
clientId && !clientId->empty());
// Cache clientId for potential refresh flow uses in the future.
oidcClientGlobalParams.oidcClientId = clientId.toString();
oidcClientGlobalParams.oidcClientId = clientId->toString();
// Construct body of POST request to device authorization endpoint based on provided
// parameters.
StringBuilder deviceCodeRequestSb;
appendPostBodyRequiredParams(&deviceCodeRequestSb, clientId);
appendPostBodyRequiredParams(&deviceCodeRequestSb, clientId.value());
appendPostBodyDeviceCodeRequestParams(&deviceCodeRequestSb, serverReply.getRequestScopes());
auto deviceCodeRequest = deviceCodeRequestSb.str();
@ -158,7 +160,7 @@ std::pair<std::string, std::string> doDeviceAuthorizationGrantFlow(
// Poll token endpoint for access and refresh tokens. It should return immediately since
// the shell blocks on the authenticationSimulator until it completes, but poll anyway.
StringBuilder tokenRequestSb;
appendPostBodyRequiredParams(&tokenRequestSb, clientId);
appendPostBodyRequiredParams(&tokenRequestSb, clientId.value());
appendPostBodyTokenRequestParams(&tokenRequestSb, deviceAuthorizationResponse.getDeviceCode());
auto tokenRequest = tokenRequestSb.str();

View File

@ -58,6 +58,7 @@ structs:
clientId:
description: "Unique client ID for this OIDC client"
type: string
optional: true # Omitted if an IdP is intended for machine flows only
requestScopes:
description: "Additional scopes to request from IDP"
type: array<string>

View File

@ -297,7 +297,7 @@ StatusWith<std::pair<ParsedCollModRequest, BSONObj>> parseCollModRequest(
}
if (coll->isCapped() &&
!feature_flags::gFeatureFlagTTLIndexesOnCappedCollections.isEnabled(
serverGlobalParams.featureCompatibility)) {
serverGlobalParams.featureCompatibility.acquireFCVSnapshot())) {
return {ErrorCodes::InvalidOptions,
"TTL indexes are not supported for capped collections."};
}
@ -502,8 +502,8 @@ StatusWith<std::pair<ParsedCollModRequest, BSONObj>> parseCollModRequest(
// (Generic FCV reference): This FCV check should exist across LTS binary versions.
multiversion::FeatureCompatibilityVersion fcv;
if (serverGlobalParams.validateFeaturesAsPrimary.load() &&
serverGlobalParams.featureCompatibility.isLessThan(multiversion::GenericFCV::kLatest,
&fcv)) {
serverGlobalParams.featureCompatibility.acquireFCVSnapshot().isLessThan(
multiversion::GenericFCV::kLatest, &fcv)) {
maxFeatureCompatibilityVersion = fcv;
}
auto validatorObj = *validator;
@ -980,7 +980,7 @@ Status _collModInternal(OperationContext* opCtx,
if (changed) {
coll.getWritableCollection(opCtx)->setTimeseriesOptions(opCtx, newOptions);
if (feature_flags::gTSBucketingParametersUnchanged.isEnabled(
serverGlobalParams.featureCompatibility)) {
serverGlobalParams.featureCompatibility.acquireFCVSnapshot())) {
coll.getWritableCollection(opCtx)->setTimeseriesBucketingParametersChanged(
opCtx, true);
};
@ -992,8 +992,9 @@ Status _collModInternal(OperationContext* opCtx,
// (Generic FCV reference): This FCV check happens whenever we upgrade to the latest
// version.
// TODO SERVER-80490: remove this check when 8.0 becomes the next LTS release.
if (auto version = serverGlobalParams.featureCompatibility.getVersion();
cmrNew.numModifications == 0 &&
const auto version =
serverGlobalParams.featureCompatibility.acquireFCVSnapshot().getVersion();
if (cmrNew.numModifications == 0 &&
(version == multiversion::GenericFCV::kUpgradingFromLastContinuousToLatest ||
version == multiversion::GenericFCV::kUpgradingFromLastLTSToLatest)) {
auto writableCollection = coll.getWritableCollection(opCtx);
@ -1007,8 +1008,7 @@ Status _collModInternal(OperationContext* opCtx,
// (Generic FCV reference): This FCV check should exist across LTS binary versions.
// TODO SERVER-80003 remove special version handling when LTS becomes 8.0.
if (cmrNew.numModifications == 0 && coll->timeseriesBucketingParametersHaveChanged() &&
serverGlobalParams.featureCompatibility.getVersion() ==
multiversion::GenericFCV::kDowngradingFromLatestToLastLTS) {
version == multiversion::GenericFCV::kDowngradingFromLatestToLastLTS) {
coll.getWritableCollection(opCtx)->setTimeseriesBucketingParametersChanged(opCtx,
boost::none);
}

View File

@ -113,7 +113,7 @@ TEST(CollectionOptions, ErrorBadMax) {
}
TEST(CollectionOptions, CappedSizeNotRoundUpForAlignment) {
serverGlobalParams.mutableFeatureCompatibility.setVersion(
serverGlobalParams.mutableFCV.setVersion(
multiversion::FeatureCompatibilityVersion::kVersion_6_2);
const long long kUnalignedCappedSize = 1000;
const long long kAlignedCappedSize = 1000;

View File

@ -948,8 +948,8 @@ Status DatabaseImpl::userCreateNS(OperationContext* opCtx,
// (Generic FCV reference): This FCV check should exist across LTS binary versions.
multiversion::FeatureCompatibilityVersion fcv;
if (serverGlobalParams.validateFeaturesAsPrimary.load() &&
serverGlobalParams.featureCompatibility.isLessThan(multiversion::GenericFCV::kLatest,
&fcv)) {
serverGlobalParams.featureCompatibility.acquireFCVSnapshot().isLessThan(
multiversion::GenericFCV::kLatest, &fcv)) {
expCtx->maxFeatureCompatibilityVersion = fcv;
}

View File

@ -355,7 +355,7 @@ void dropReadyIndexes(OperationContext* opCtx,
const auto& shardKey = collDescription.getShardKeyPattern();
const bool skipDropIndex = skipDroppingHashedShardKeyIndex ||
!(gFeatureFlagShardKeyIndexOptionalHashedSharding.isEnabled(
serverGlobalParams.featureCompatibility) &&
serverGlobalParams.featureCompatibility.acquireFCVSnapshot()) &&
shardKey.isHashedPattern());
if (isCompatibleWithShardKey(opCtx,
CollectionPtr(collection),

View File

@ -275,7 +275,7 @@ void IndexBuildBlock::success(OperationContext* opCtx, Collection* collection) {
// collection scan, which does not use an index.
if (spec.hasField(IndexDescriptor::kExpireAfterSecondsFieldName) &&
(feature_flags::gFeatureFlagTTLIndexesOnCappedCollections.isEnabled(
serverGlobalParams.featureCompatibility) ||
serverGlobalParams.featureCompatibility.acquireFCVSnapshot()) ||
!coll->isCapped())) {
auto validateStatus = index_key_validate::validateExpireAfterSeconds(
spec[IndexDescriptor::kExpireAfterSecondsFieldName],

View File

@ -275,7 +275,7 @@ void IndexCatalogImpl::init(OperationContext* opCtx,
// Note that TTL deletion is supported on capped clustered collections via bounded
// collection scan, which does not use an index.
if (feature_flags::gFeatureFlagTTLIndexesOnCappedCollections.isEnabled(
serverGlobalParams.featureCompatibility) ||
serverGlobalParams.featureCompatibility.acquireFCVSnapshot()) ||
!collection->isCapped()) {
if (opCtx->lockState()->inAWriteUnitOfWork()) {
opCtx->recoveryUnit()->onCommit(
@ -519,7 +519,7 @@ StatusWith<BSONObj> IndexCatalogImpl::prepareSpecForCreate(
if (collection && collection->isCapped() &&
validatedSpec.hasField(IndexDescriptor::kExpireAfterSecondsFieldName) &&
!feature_flags::gFeatureFlagTTLIndexesOnCappedCollections.isEnabled(
serverGlobalParams.featureCompatibility) &&
serverGlobalParams.featureCompatibility.acquireFCVSnapshot()) &&
MONGO_likely(!ignoreTTLIndexCappedCollectionCheck.shouldFail())) {
return {ErrorCodes::CannotCreateIndex, "Cannot create TTL index on a capped collection"};
}
@ -975,15 +975,15 @@ Status IndexCatalogImpl::_isSpecOk(OperationContext* opCtx,
return wildcardSpecStatus;
}
} else if (pluginName == IndexNames::COLUMN) {
const auto fcvSnapshot = serverGlobalParams.featureCompatibility.acquireFCVSnapshot();
uassert(ErrorCodes::NotImplemented,
str::stream() << pluginName
<< " indexes are under development and cannot be used without "
"enabling the feature flag",
// With our testing failpoint we may try to run this code before we've initialized
// the FCV.
!serverGlobalParams.featureCompatibility.isVersionInitialized() ||
feature_flags::gFeatureFlagColumnstoreIndexes.isEnabled(
serverGlobalParams.featureCompatibility));
!fcvSnapshot.isVersionInitialized() ||
feature_flags::gFeatureFlagColumnstoreIndexes.isEnabled(fcvSnapshot));
if (auto columnSpecStatus = validateColumnStoreSpec(collection, spec, indexVersion);
!columnSpecStatus.isOK()) {
return columnSpecStatus;
@ -1740,7 +1740,7 @@ Status IndexCatalogImpl::_updateRecord(OperationContext* const opCtx,
// index has incorrect keys. Replace this failpoint with a test command instead.
if (auto failpoint = skipUpdatingIndexDocument.scoped(); MONGO_unlikely(failpoint.isActive()) &&
repl::feature_flags::gSecondaryIndexChecksInDbCheck.isEnabled(
serverGlobalParams.featureCompatibility)) {
serverGlobalParams.featureCompatibility.acquireFCVSnapshot())) {
auto indexName = failpoint.getData()["indexName"].valueStringDataSafe();
if (indexName == index->descriptor()->indexName()) {
LOGV2_DEBUG(

View File

@ -169,7 +169,7 @@ auto makeOnSuppressedErrorFn(const std::function<void()>& saveCursorBeforeWrite,
bool shouldRelaxConstraints(OperationContext* opCtx, const CollectionPtr& collection) {
if (!feature_flags::gIndexBuildGracefulErrorHandling.isEnabled(
serverGlobalParams.featureCompatibility)) {
serverGlobalParams.featureCompatibility.acquireFCVSnapshot())) {
// Always suppress.
return true;
}

View File

@ -136,7 +136,7 @@ ValidateState::ValidateState(OperationContext* opCtx,
if (additionalOptions.enforceTimeseriesBucketsAreAlwaysCompressed) {
if (TestingProctor::instance().isEnabled() &&
feature_flags::gTimeseriesAlwaysUseCompressedBuckets.isEnabled(
serverGlobalParams.featureCompatibility)) {
serverGlobalParams.featureCompatibility.acquireFCVSnapshot())) {
_enforceTimeseriesBucketsAreAlwaysCompressed = true;
} else {
LOGV2_WARNING(7735102, "Not enforcing that time-series buckets are always compressed");

View File

@ -138,7 +138,7 @@ std::unique_ptr<PlanExecutor, PlanExecutor::Deleter> getDeleteExpiredPreImagesEx
bool useUnreplicatedTruncates() {
bool res = feature_flags::gFeatureFlagUseUnreplicatedTruncatesForDeletions.isEnabled(
serverGlobalParams.featureCompatibility);
serverGlobalParams.featureCompatibility.acquireFCVSnapshot());
return res;
}
} // namespace

View File

@ -68,7 +68,7 @@ namespace change_stream_serverless_helpers {
namespace {
bool isServerlessChangeStreamFeatureFlagEnabled() {
return feature_flags::gFeatureFlagServerlessChangeStreams.isEnabled(
serverGlobalParams.featureCompatibility);
serverGlobalParams.featureCompatibility.acquireFCVSnapshot());
}
} // namespace

View File

@ -371,7 +371,7 @@ StatusWith<std::vector<BSONObj>> DefaultClonerImpl::_filterCollectionsForClone(
const auto nss = NamespaceStringUtil::deserialize(fromDBName, collectionName.c_str());
if (nss.isSystem()) {
if (!nss.isLegalClientSystemNS(serverGlobalParams.featureCompatibility)) {
if (!nss.isLegalClientSystemNS()) {
LOGV2_DEBUG(20419, 2, "\t\t not cloning because system collection");
continue;
}

View File

@ -165,7 +165,7 @@ public:
uassert(6660400,
"Analyze command requires common query framework feature flag to be enabled",
feature_flags::gFeatureFlagCommonQueryFramework.isEnabled(
serverGlobalParams.featureCompatibility));
serverGlobalParams.featureCompatibility.acquireFCVSnapshot()));
const auto& cmd = request();
const NamespaceString& nss = ns();

View File

@ -1262,10 +1262,10 @@ public:
const Command* command,
const OpMsgRequest& opMsgRequest)
: InvocationBaseGen(opCtx, command, opMsgRequest), _commandObj(opMsgRequest.body) {
uassert(
ErrorCodes::CommandNotSupported,
"BulkWrite may not be run without featureFlagBulkWriteCommand enabled",
gFeatureFlagBulkWriteCommand.isEnabled(serverGlobalParams.featureCompatibility));
uassert(ErrorCodes::CommandNotSupported,
"BulkWrite may not be run without featureFlagBulkWriteCommand enabled",
gFeatureFlagBulkWriteCommand.isEnabled(
serverGlobalParams.featureCompatibility.acquireFCVSnapshot()));
bulk_write_common::validateRequest(request(), /*isRouter=*/false);

View File

@ -562,7 +562,7 @@ CreateIndexesReply runCreateIndexesWithCoordinator(OperationContext* opCtx,
!opCtx->inMultiDocumentTransaction());
if (feature_flags::gIndexBuildGracefulErrorHandling.isEnabled(
serverGlobalParams.featureCompatibility)) {
serverGlobalParams.featureCompatibility.acquireFCVSnapshot())) {
uassertStatusOK(IndexBuildsCoordinator::checkDiskSpaceSufficientToStartIndexBuild(opCtx));
}

View File

@ -109,6 +109,7 @@ MONGO_FAIL_POINT_DEFINE(hangBeforeExtraIndexKeysHashing);
namespace mongo {
MONGO_FAIL_POINT_DEFINE(hangBeforeDbCheckLogOp);
MONGO_FAIL_POINT_DEFINE(hangBeforeProcessingDbCheckRun);
MONGO_FAIL_POINT_DEFINE(hangBeforeProcessingFirstBatch);
@ -230,7 +231,7 @@ std::unique_ptr<DbCheckRun> singleCollectionRun(OperationContext* opCtx,
const DbCheckSingleInvocation& invocation) {
const auto gSecondaryIndexChecksInDbCheck =
repl::feature_flags::gSecondaryIndexChecksInDbCheck.isEnabled(
serverGlobalParams.featureCompatibility);
serverGlobalParams.featureCompatibility.acquireFCVSnapshot());
if (!gSecondaryIndexChecksInDbCheck) {
uassert(ErrorCodes::InvalidOptions,
"When featureFlagSecondaryIndexChecksInDbCheck is not enabled, the validateMode "
@ -778,19 +779,17 @@ Status DbChecker::_hashExtraIndexKeysCheck(OperationContext* opCtx,
// longer than the time it takes between starting and replicating a batch on the
// primary. Otherwise, the readTimestamp will not be available on a secondary by the
// time it processes the oplog entry.
opCtx->recoveryUnit()->setTimestampReadSource(RecoveryUnit::ReadSource::kNoOverlap);
const auto readSource = ReadSourceWithTimestamp{RecoveryUnit::ReadSource::kNoOverlap};
// dbCheck writes to the oplog, so we need to take an IX global lock. We don't need to write
// to the collection, however, so we use acquireCollectionMaybeLockFree with a read
// acquisition request.
Lock::GlobalLock glob(opCtx, MODE_IX);
const CollectionAcquisition collAcquisition = acquireCollectionMaybeLockFree(
const DbCheckAcquisition acquisition(
opCtx,
CollectionAcquisitionRequest::fromOpCtx(
opCtx, _info.nss, AcquisitionPrerequisites::OperationType::kRead));
if (!collAcquisition.exists() ||
collAcquisition.getCollectionPtr().get()->uuid() != _info.uuid) {
_info.nss,
readSource,
// On the primary we must always block on prepared updates to guarantee snapshot isolation.
PrepareConflictBehavior::kEnforce);
if (!acquisition.coll.exists() ||
acquisition.coll.getCollectionPtr().get()->uuid() != _info.uuid) {
Status status = Status(ErrorCodes::IndexNotFound,
str::stream() << "cannot find collection for ns "
<< _info.nss.toStringForErrorMsg() << " and uuid "
@ -808,7 +807,7 @@ Status DbChecker::_hashExtraIndexKeysCheck(OperationContext* opCtx,
return status;
}
const CollectionPtr& collection = collAcquisition.getCollectionPtr();
const CollectionPtr& collection = acquisition.coll.getCollectionPtr();
// TODO SERVER-80347: Add check for stepdown here.
auto readTimestamp = opCtx->recoveryUnit()->getPointInTimeReadTimestamp(opCtx);
@ -848,7 +847,7 @@ Status DbChecker::_hashExtraIndexKeysCheck(OperationContext* opCtx,
boost::optional<DbCheckHasher> hasher;
try {
hasher.emplace(opCtx,
collection,
acquisition,
firstBson,
lastBson,
_info.secondaryIndexCheckParameters,
@ -1531,29 +1530,27 @@ StatusWith<DbCheckCollectionBatchStats> DbChecker::_runBatch(OperationContext* o
// longer than the time it takes between starting and replicating a batch on the
// primary. Otherwise, the readTimestamp will not be available on a secondary by the
// time it processes the oplog entry.
opCtx->recoveryUnit()->setTimestampReadSource(RecoveryUnit::ReadSource::kNoOverlap);
const auto readSource = ReadSourceWithTimestamp{RecoveryUnit::ReadSource::kNoOverlap};
// dbCheck writes to the oplog, so we need to take an IX global lock. We don't need to
// write to the collection, however, so we use acquireCollectionMaybeLockFree with a
// read acquisition request.
Lock::GlobalLock glob(opCtx, MODE_IX);
const CollectionAcquisition collAcquisition = acquireCollectionMaybeLockFree(
// Acquires locks and sets appropriate state on the RecoveryUnit.
const DbCheckAcquisition acquisition(
opCtx,
CollectionAcquisitionRequest::fromOpCtx(
opCtx, _info.nss, AcquisitionPrerequisites::OperationType::kRead));
_info.nss,
readSource,
// On the primary we must always block on prepared updates to guarantee snapshot isolation.
PrepareConflictBehavior::kEnforce);
if (_stepdownHasOccurred(opCtx, _info.nss)) {
_done = true;
return Status(ErrorCodes::PrimarySteppedDown, "dbCheck terminated due to stepdown");
}
if (!collAcquisition.exists()) {
if (!acquisition.coll.exists()) {
const auto msg = "Collection under dbCheck no longer exists";
return {ErrorCodes::NamespaceNotFound, msg};
}
// The CollectionPtr needs to outlive the DbCheckHasher as it's used internally.
const CollectionPtr& collectionPtr = collAcquisition.getCollectionPtr();
const CollectionPtr& collectionPtr = acquisition.coll.getCollectionPtr();
if (collectionPtr.get()->uuid() != _info.uuid) {
const auto msg = "Collection under dbCheck no longer exists";
return {ErrorCodes::NamespaceNotFound, msg};
@ -1567,7 +1564,7 @@ StatusWith<DbCheckCollectionBatchStats> DbChecker::_runBatch(OperationContext* o
Status status = Status::OK();
try {
hasher.emplace(opCtx,
collectionPtr,
acquisition,
first,
_info.end,
_info.secondaryIndexCheckParameters,
@ -1619,6 +1616,11 @@ StatusWith<DbCheckCollectionBatchStats> DbChecker::_runBatch(OperationContext* o
batch.setMaxKey(BSONKey::parseFromBSON(hasher->lastKey().firstElement()));
}
if (MONGO_unlikely(hangBeforeDbCheckLogOp.shouldFail())) {
LOGV2(8230500, "Hanging dbcheck due to failpoint 'hangBeforeDbCheckLogOp'");
hangBeforeDbCheckLogOp.pauseWhileSet();
}
// Send information on this batch over the oplog.
DbCheckCollectionBatchStats result;
result.logToHealthLog = _shouldLogBatch(batch);

View File

@ -354,7 +354,8 @@ void FeatureCompatibilityVersion::validateSetFeatureCompatibilityVersionRequest(
uassert(5563601,
"Cannot transition to fully upgraded or fully downgraded state if the shard is not "
"in kUpgrading or kDowngrading state",
serverGlobalParams.featureCompatibility.isUpgradingOrDowngrading());
serverGlobalParams.featureCompatibility.acquireFCVSnapshot()
.isUpgradingOrDowngrading());
tassert(5563502,
"Shard received a request for phase 2 of the 'setFeatureCompatibilityVersion' "
@ -388,7 +389,8 @@ void FeatureCompatibilityVersion::updateFeatureCompatibilityVersionDocument(
// Only transition to fully upgraded or downgraded states when we have completed all required
// upgrade/downgrade behavior, unless it is the newly added downgrading to upgrading path.
auto transitioningVersion = setTargetVersion &&
serverGlobalParams.featureCompatibility.isUpgradingOrDowngrading(fromVersion) &&
serverGlobalParams.featureCompatibility.acquireFCVSnapshot().isUpgradingOrDowngrading(
fromVersion) &&
!(fromVersion == GenericFCV::kDowngradingFromLatestToLastLTS &&
newVersion == GenericFCV::kLatest)
? fromVersion
@ -521,7 +523,8 @@ bool FeatureCompatibilityVersion::hasNoReplicatedCollections(OperationContext* o
void FeatureCompatibilityVersion::updateMinWireVersion(OperationContext* opCtx) {
WireSpec& wireSpec = WireSpec::getWireSpec(opCtx->getServiceContext());
const auto currentFcv = serverGlobalParams.featureCompatibility.getVersion();
const auto fcvSnapshot = serverGlobalParams.featureCompatibility.acquireFCVSnapshot();
const auto currentFcv = fcvSnapshot.getVersion();
// The reason we set the minWireVersion to LATEST_WIRE_VERSION when downgrading from latest as
// well as on upgrading to latest is because we shouldnt decrease the minWireVersion until we
// have fully downgraded to the lower FCV in case we get any backwards compatibility breakages,
@ -530,7 +533,7 @@ void FeatureCompatibilityVersion::updateMinWireVersion(OperationContext* opCtx)
// communicate with downgraded binary nodes until the FCV is completely downgraded to
// `kVersion_Y`.
if (currentFcv == GenericFCV::kLatest ||
(serverGlobalParams.featureCompatibility.isUpgradingOrDowngrading() &&
(fcvSnapshot.isUpgradingOrDowngrading() &&
currentFcv != GenericFCV::kUpgradingFromLastLTSToLastContinuous)) {
// FCV == kLatest or FCV is upgrading/downgrading to or from kLatest.
WireSpec::Specification newSpec = *wireSpec.get();
@ -558,7 +561,8 @@ void FeatureCompatibilityVersion::initializeForStartup(OperationContext* opCtx)
invariant(opCtx->lockState()->isW());
auto featureCompatibilityVersion = findFeatureCompatibilityVersionDocument(opCtx);
if (!featureCompatibilityVersion) {
serverGlobalParams.featureCompatibility.logFCVWithContext("startup"_sd);
serverGlobalParams.featureCompatibility.acquireFCVSnapshot().logFCVWithContext(
"startup"_sd);
return;
}
@ -582,13 +586,14 @@ void FeatureCompatibilityVersion::initializeForStartup(OperationContext* opCtx)
}
auto version = swVersion.getValue();
serverGlobalParams.mutableFeatureCompatibility.setVersion(version);
serverGlobalParams.mutableFCV.setVersion(version);
FeatureCompatibilityVersion::updateMinWireVersion(opCtx);
const auto fcvSnapshot = serverGlobalParams.featureCompatibility.acquireFCVSnapshot();
serverGlobalParams.featureCompatibility.logFCVWithContext("startup"_sd);
fcvSnapshot.logFCVWithContext("startup"_sd);
// On startup, if the version is in an upgrading or downgrading state, print a warning.
if (serverGlobalParams.featureCompatibility.isUpgradingOrDowngrading()) {
if (fcvSnapshot.isUpgradingOrDowngrading()) {
LOGV2_WARNING_OPTIONS(
4978301,
{logv2::LogTag::kStartupWarnings},
@ -637,7 +642,8 @@ void FeatureCompatibilityVersion::fassertInitializedAfterStartup(OperationContex
// startup. In standalone mode, FCV is initialized during startup, even in read-only mode.
bool isWriteableStorageEngine = storageGlobalParams.engine != "devnull";
if (isWriteableStorageEngine && (!usingReplication || nonLocalDatabases)) {
invariant(serverGlobalParams.featureCompatibility.isVersionInitialized());
invariant(
serverGlobalParams.featureCompatibility.acquireFCVSnapshot().isVersionInitialized());
}
}
@ -667,12 +673,13 @@ void FeatureCompatibilityVersionParameter::append(OperationContext* opCtx,
BSONObjBuilder* b,
StringData name,
const boost::optional<TenantId>&) {
const auto fcvSnapshot = serverGlobalParams.featureCompatibility.acquireFCVSnapshot();
uassert(ErrorCodes::UnknownFeatureCompatibilityVersion,
str::stream() << name << " is not yet known.",
serverGlobalParams.featureCompatibility.isVersionInitialized());
fcvSnapshot.isVersionInitialized());
BSONObjBuilder featureCompatibilityVersionBuilder(b->subobjStart(name));
auto version = serverGlobalParams.featureCompatibility.getVersion();
auto version = fcvSnapshot.getVersion();
FeatureCompatibilityVersionDocument fcvDoc = fcvTransitions.getFCVDocument(version);
featureCompatibilityVersionBuilder.appendElements(fcvDoc.toBSON().removeField("_id"));
if (!fcvDoc.getTargetVersion()) {
@ -718,16 +725,21 @@ FixedFCVRegion::FixedFCVRegion(OperationContext* opCtx)
FixedFCVRegion::~FixedFCVRegion() = default;
const ServerGlobalParams::FeatureCompatibility& FixedFCVRegion::operator*() const {
// Note that the FixedFCVRegion only prevents the on-disk FCV from changing, not
// the in-memory FCV. (which for example could be reset during initial sync). The operator* and
// operator-> functions return a MutableFCV, which could change at different points in time. If you
// wanted to get a consistent snapshot of the in-memory FCV, you should still use the
// ServerGlobalParams::MutableFCV's acquireFCVSnapshot() function to get a FCVSnapshot.
const ServerGlobalParams::MutableFCV& FixedFCVRegion::operator*() const {
return serverGlobalParams.featureCompatibility;
}
const ServerGlobalParams::FeatureCompatibility* FixedFCVRegion::operator->() const {
const ServerGlobalParams::MutableFCV* FixedFCVRegion::operator->() const {
return &serverGlobalParams.featureCompatibility;
}
bool FixedFCVRegion::operator==(const FCV& other) const {
return serverGlobalParams.featureCompatibility.getVersion() == other;
return serverGlobalParams.featureCompatibility.acquireFCVSnapshot().getVersion() == other;
}
bool FixedFCVRegion::operator!=(const FCV& other) const {

View File

@ -144,7 +144,13 @@ public:
};
/**
* Utility class to prevent the FCV from changing while the FixedFCVRegion is in scope.
* Utility class to prevent the on-disk FCV from changing while the FixedFCVRegion is in scope.
*
* Note that this does not prevent the in-memory FCV from changing (which for example could be reset
* during initial sync). The operator* and operator-> functions return a MutableFCV, which could
* change at different points in time, so if you wanted to get a consistent snapshot of the
* in-memory FCV, you should still use the ServerGlobalParams::MutableFCV's acquireFCVSnapshot()
* function.
*/
class FixedFCVRegion {
public:
@ -154,8 +160,8 @@ public:
bool operator==(const multiversion::FeatureCompatibilityVersion& other) const;
bool operator!=(const multiversion::FeatureCompatibilityVersion& other) const;
const ServerGlobalParams::FeatureCompatibility& operator*() const;
const ServerGlobalParams::FeatureCompatibility* operator->() const;
const ServerGlobalParams::MutableFCV& operator*() const;
const ServerGlobalParams::MutableFCV* operator->() const;
private:
Lock::SharedLock _lk;

View File

@ -202,7 +202,7 @@ query_settings::QuerySettings lookupQuerySettingsForFind(
const NamespaceString& nss) {
// No QuerySettings lookup for IDHACK queries.
if (!feature_flags::gFeatureFlagQuerySettings.isEnabled(
serverGlobalParams.featureCompatibility) ||
serverGlobalParams.featureCompatibility.acquireFCVSnapshot()) ||
(collection &&
isIdHackEligibleQuery(
collection, *parsedRequest.findCommandRequest, parsedRequest.collator.get()))) {

View File

@ -150,7 +150,8 @@ CleanupStats cleanupEncryptedCollection(OperationContext* opCtx,
uassert(7618803,
str::stream() << "Feature flag `FLE2CleanupCommand` must be enabled to run "
<< CleanupStructuredEncryptionData::kCommandName,
gFeatureFlagFLE2CleanupCommand.isEnabled(serverGlobalParams.featureCompatibility));
gFeatureFlagFLE2CleanupCommand.isEnabled(
serverGlobalParams.featureCompatibility.acquireFCVSnapshot()));
uassert(7618804,
str::stream() << CleanupStructuredEncryptionData::kCommandName

View File

@ -84,7 +84,7 @@ public:
Reply typedRun(OperationContext* opCtx) {
if (!feature_flags::gFeatureFlagAuditConfigClusterParameter.isEnabled(
serverGlobalParams.featureCompatibility)) {
serverGlobalParams.featureCompatibility.acquireFCVSnapshot())) {
uassert(ErrorCodes::IllegalOperation,
str::stream() << Request::kCommandName << " cannot be run on standalones",
repl::ReplicationCoordinator::get(opCtx)->getSettings().isReplSet());

View File

@ -259,7 +259,7 @@ public:
uassert(7746400,
"setQuerySettings command is unknown",
feature_flags::gFeatureFlagQuerySettings.isEnabled(
serverGlobalParams.featureCompatibility));
serverGlobalParams.featureCompatibility.acquireFCVSnapshot()));
auto response =
stdx::visit(OverloadedVisitor{
[&](const query_shape::QueryShapeHash& queryShapeHash) {
@ -323,7 +323,7 @@ public:
uassert(7746700,
"removeQuerySettings command is unknown",
feature_flags::gFeatureFlagQuerySettings.isEnabled(
serverGlobalParams.featureCompatibility));
serverGlobalParams.featureCompatibility.acquireFCVSnapshot()));
auto tenantId = request().getDbName().tenantId();
auto queryShapeHash =
stdx::visit(OverloadedVisitor{

View File

@ -87,7 +87,7 @@ void setClusterParameterImplShard(OperationContext* opCtx,
(serverGlobalParams.clusterRole.has(ClusterRole::None)));
if (!feature_flags::gFeatureFlagAuditConfigClusterParameter.isEnabled(
serverGlobalParams.featureCompatibility)) {
serverGlobalParams.featureCompatibility.acquireFCVSnapshot())) {
uassert(ErrorCodes::IllegalOperation,
str::stream() << SetClusterParameter::kCommandName
<< " cannot be run on standalones",

View File

@ -316,7 +316,8 @@ public:
Lock::ExclusiveLock setFCVCommandLock(opCtx, commandMutex);
const auto requestedVersion = request.getCommandParameter();
const auto actualVersion = serverGlobalParams.featureCompatibility.getVersion();
const auto actualVersion =
serverGlobalParams.featureCompatibility.acquireFCVSnapshot().getVersion();
auto isConfirmed = request.getConfirm().value_or(false);
const auto upgradeMsg =
@ -466,7 +467,8 @@ public:
}
}
invariant(serverGlobalParams.featureCompatibility.isUpgradingOrDowngrading());
invariant(serverGlobalParams.featureCompatibility.acquireFCVSnapshot()
.isUpgradingOrDowngrading());
if (!request.getPhase() || request.getPhase() == SetFCVPhaseEnum::kPrepare) {
if (serverGlobalParams.clusterRole.has(ClusterRole::ConfigServer)) {
@ -512,7 +514,8 @@ public:
}
}
invariant(serverGlobalParams.featureCompatibility.isUpgradingOrDowngrading());
invariant(serverGlobalParams.featureCompatibility.acquireFCVSnapshot()
.isUpgradingOrDowngrading());
invariant(!request.getPhase() || request.getPhase() == SetFCVPhaseEnum::kComplete);
// All feature-specific FCV upgrade or downgrade code should go into the respective
@ -538,7 +541,7 @@ public:
FeatureCompatibilityVersion::updateFeatureCompatibilityVersionDocument(
opCtx,
serverGlobalParams.featureCompatibility.getVersion(),
serverGlobalParams.featureCompatibility.acquireFCVSnapshot().getVersion(),
requestedVersion,
isFromConfigServer,
changeTimestamp,
@ -579,9 +582,9 @@ private:
// We do not expect any other feature-specific work to be done in the 'start' phase.
void _shardServerPhase1Tasks(OperationContext* opCtx,
multiversion::FeatureCompatibilityVersion requestedVersion) {
invariant(serverGlobalParams.featureCompatibility.isUpgradingOrDowngrading());
const auto& [originalVersion, _] =
getTransitionFCVFromAndTo(serverGlobalParams.featureCompatibility.getVersion());
const auto fcvSnapshot = serverGlobalParams.featureCompatibility.acquireFCVSnapshot();
invariant(fcvSnapshot.isUpgradingOrDowngrading());
const auto& [originalVersion, _] = getTransitionFCVFromAndTo(fcvSnapshot.getVersion());
const auto isDowngrading = originalVersion > requestedVersion;
const auto isUpgrading = originalVersion < requestedVersion;
@ -757,8 +760,8 @@ private:
void _maybeMigrateAuditConfig(OperationContext* opCtx,
const multiversion::FeatureCompatibilityVersion requestedVersion,
boost::optional<Timestamp> changeTimestamp) {
const auto& [fromVersion, _] =
getTransitionFCVFromAndTo(serverGlobalParams.featureCompatibility.getVersion());
const auto& [fromVersion, _] = getTransitionFCVFromAndTo(
serverGlobalParams.featureCompatibility.acquireFCVSnapshot().getVersion());
if (feature_flags::gFeatureFlagAuditConfigClusterParameter
.isEnabledOnTargetFCVButDisabledOnOriginalFCV(requestedVersion, fromVersion) &&
audit::migrateOldToNew) {
@ -833,8 +836,8 @@ private:
OperationContext* opCtx, const multiversion::FeatureCompatibilityVersion requestedVersion) {
// There is no need to re-create this index on upgrade, as the index is no longer
// needed to ensure resharding operations are unique.
const auto& [fromVersion, _] =
getTransitionFCVFromAndTo(serverGlobalParams.featureCompatibility.getVersion());
const auto& [fromVersion, _] = getTransitionFCVFromAndTo(
serverGlobalParams.featureCompatibility.acquireFCVSnapshot().getVersion());
if (resharding::gFeatureFlagReshardingImprovements
.isEnabledOnTargetFCVButDisabledOnOriginalFCV(requestedVersion, fromVersion)) {
LOGV2(7760401,
@ -866,7 +869,8 @@ private:
const multiversion::FeatureCompatibilityVersion requestedVersion,
const NamespaceString& indexCatalogNss) {
// TODO SERVER-67392: Remove once gGlobalIndexesShardingCatalog is enabled.
const auto actualVersion = serverGlobalParams.featureCompatibility.getVersion();
const auto actualVersion =
serverGlobalParams.featureCompatibility.acquireFCVSnapshot().getVersion();
if (feature_flags::gGlobalIndexesShardingCatalog
.isEnabledOnTargetFCVButDisabledOnOriginalFCV(requestedVersion, actualVersion)) {
uassertStatusOK(
@ -1003,9 +1007,9 @@ private:
// manually clean up some user data in order to retry the FCV downgrade.
void _userCollectionsUassertsForDowngrade(
OperationContext* opCtx, const multiversion::FeatureCompatibilityVersion requestedVersion) {
invariant(serverGlobalParams.featureCompatibility.isUpgradingOrDowngrading());
const auto& [originalVersion, _] =
getTransitionFCVFromAndTo(serverGlobalParams.featureCompatibility.getVersion());
const auto fcvSnapshot = serverGlobalParams.featureCompatibility.acquireFCVSnapshot();
invariant(fcvSnapshot.isUpgradingOrDowngrading());
const auto& [originalVersion, _] = getTransitionFCVFromAndTo(fcvSnapshot.getVersion());
if (feature_flags::gFeatureFlagAuditConfigClusterParameter
.isDisabledOnTargetFCVButEnabledOnOriginalFCV(requestedVersion, originalVersion)) {
@ -1066,9 +1070,9 @@ private:
// requestedVersion.
void _cleanUpClusterParameters(
OperationContext* opCtx, const multiversion::FeatureCompatibilityVersion requestedVersion) {
invariant(serverGlobalParams.featureCompatibility.isUpgradingOrDowngrading());
const auto& [fromVersion, _] =
getTransitionFCVFromAndTo(serverGlobalParams.featureCompatibility.getVersion());
const auto fcvSnapshot = serverGlobalParams.featureCompatibility.acquireFCVSnapshot();
invariant(fcvSnapshot.isUpgradingOrDowngrading());
const auto& [fromVersion, _] = getTransitionFCVFromAndTo(fcvSnapshot.getVersion());
auto* clusterParameters = ServerParameterSet::getClusterParameterSet();
std::vector<write_ops::DeleteOpEntry> deletes;
@ -1089,9 +1093,9 @@ private:
void _updateAuditConfigOnDowngrade(
OperationContext* opCtx, const multiversion::FeatureCompatibilityVersion requestedVersion) {
invariant(serverGlobalParams.featureCompatibility.isUpgradingOrDowngrading());
const auto& [fromVersion, _] =
getTransitionFCVFromAndTo(serverGlobalParams.featureCompatibility.getVersion());
const auto fcvSnapshot = serverGlobalParams.featureCompatibility.acquireFCVSnapshot();
invariant(fcvSnapshot.isUpgradingOrDowngrading());
const auto& [fromVersion, _] = getTransitionFCVFromAndTo(fcvSnapshot.getVersion());
if (feature_flags::gFeatureFlagAuditConfigClusterParameter
.isDisabledOnTargetFCVButEnabledOnOriginalFCV(requestedVersion, fromVersion)) {
@ -1121,9 +1125,9 @@ private:
// they would turn into a Support case.
void _internalServerCleanupForDowngrade(
OperationContext* opCtx, const multiversion::FeatureCompatibilityVersion requestedVersion) {
invariant(serverGlobalParams.featureCompatibility.isUpgradingOrDowngrading());
const auto& [originalVersion, _] =
getTransitionFCVFromAndTo(serverGlobalParams.featureCompatibility.getVersion());
const auto fcvSnapshot = serverGlobalParams.featureCompatibility.acquireFCVSnapshot();
invariant(fcvSnapshot.isUpgradingOrDowngrading());
const auto& [originalVersion, _] = getTransitionFCVFromAndTo(fcvSnapshot.getVersion());
if (serverGlobalParams.clusterRole.has(ClusterRole::ShardServer) ||
serverGlobalParams.clusterRole.has(ClusterRole::None)) {
@ -1393,7 +1397,8 @@ private:
const SetFeatureCompatibilityVersion& request,
boost::optional<Timestamp> changeTimestamp) {
const auto requestedVersion = request.getCommandParameter();
const auto actualVersion = serverGlobalParams.featureCompatibility.getVersion();
const auto actualVersion =
serverGlobalParams.featureCompatibility.acquireFCVSnapshot().getVersion();
auto isFromConfigServer = request.getFromConfigServer().value_or(false);
hangDowngradingBeforeIsCleaningServerMetadata.pauseWhileSet(opCtx);
@ -1452,7 +1457,8 @@ private:
*/
void _cancelServerlessMigrations(OperationContext* opCtx) {
invariant(repl::ReplicationCoordinator::get(opCtx)->getSettings().isServerless());
invariant(serverGlobalParams.featureCompatibility.isUpgradingOrDowngrading());
invariant(serverGlobalParams.featureCompatibility.acquireFCVSnapshot()
.isUpgradingOrDowngrading());
auto donorService = checked_cast<TenantMigrationDonorService*>(
repl::PrimaryOnlyServiceRegistry::get(opCtx->getServiceContext())
->lookupServiceByName(TenantMigrationDonorService::kServiceName));

View File

@ -104,7 +104,8 @@ public:
uassert(
5356100,
"donorStartMigration not available while upgrading or downgrading the donor FCV",
!serverGlobalParams.featureCompatibility.isUpgradingOrDowngrading());
!serverGlobalParams.featureCompatibility.acquireFCVSnapshot()
.isUpgradingOrDowngrading());
uassert(ErrorCodes::IllegalOperation,
"tenant migrations are only available if --serverless is enabled",

View File

@ -107,7 +107,8 @@ public:
uassert(
5356101,
"recipientSyncData not available while upgrading or downgrading the recipient FCV",
!serverGlobalParams.featureCompatibility.isUpgradingOrDowngrading());
!serverGlobalParams.featureCompatibility.acquireFCVSnapshot()
.isUpgradingOrDowngrading());
const auto& cmd = request();
const auto migrationProtocol = cmd.getProtocol().value_or(kDefaultMigrationProtocol);

View File

@ -336,6 +336,8 @@ public:
boost::optional<std::size_t> queryStatsKeyHash;
// The Key used by query stats to generate the query stats store key.
std::unique_ptr<query_stats::Key> queryStatsKey;
// True if the request was rate limited and stats should not be collected.
bool queryStatsRateLimited{false};
// The query framework that this operation used. Will be unknown for non query operations.
PlanExecutor::QueryFramework queryFramework{PlanExecutor::QueryFramework::kUnknown};

View File

@ -215,16 +215,16 @@ PlanStage::StageState CollectionScan::doWork(WorkingSetID* out) {
}
}
if (_params.resumeAfterRecordId && !_params.resumeAfterRecordId->isNull()) {
if (_params.resumeAfterRecordId) {
invariant(!_params.tailable);
invariant(_lastSeenId.isNull());
// Seek to where we are trying to resume the scan from. Signal a KeyNotFound
// error if the record no longer exists.
// error if the record no longer exists or if the recordId is null.
//
// Note that we want to return the record *after* this one since we have already
// returned this one prior to the resume.
auto& recordIdToSeek = *_params.resumeAfterRecordId;
if (!_cursor->seekExact(recordIdToSeek)) {
if (recordIdToSeek.isNull() || !_cursor->seekExact(recordIdToSeek)) {
uasserted(ErrorCodes::KeyNotFound,
str::stream()
<< "Failed to resume collection scan: the recordId from "
@ -373,7 +373,7 @@ BSONObj CollectionScan::getPostBatchResumeToken() const {
BSONObjBuilder builder;
_lastSeenId.serializeToken("$recordId", &builder);
if (resharding::gFeatureFlagReshardingImprovements.isEnabled(
serverGlobalParams.featureCompatibility)) {
serverGlobalParams.featureCompatibility.acquireFCVSnapshot())) {
auto initialSyncId =
repl::ReplicationCoordinator::get(opCtx())->getInitialSyncId(opCtx());
if (initialSyncId) {

View File

@ -338,13 +338,18 @@ PlanStage::StageState DeleteStage::doWork(WorkingSetID* out) {
});
if (restoreStateRet != PlanStage::NEED_TIME) {
if (restoreStateRet == PlanStage::NEED_YIELD && stageIsEOF) {
if (restoreStateRet == PlanStage::NEED_YIELD && stageIsEOF &&
!opCtx()->lockState()->inAWriteUnitOfWork()) {
// If this stage is already exhausted it won't use its children stages anymore and
// therefore it's okay if we failed to restore them. Avoid requesting a yield to the
// plan executor. Restoring from yield could fail due to a sharding placement change.
// Throwing a StaleConfig error is undesirable after an "delete one" operation has
// already performed a write because the router would retry. Unset _idReturning as we'll
// return the document in this stage iteration.
//
// If this plan is part of a larger encompassing WUOW it would be illegal to skip
// returning NEED_YIELD, so we don't skip it. In this case, such as multi-doc
// transactions, this is okay as the PlanExecutor is not allowed to auto-yield.
_idReturning = WorkingSet::INVALID_ID;
} else {
return restoreStateRet;

View File

@ -439,6 +439,7 @@ Document DocumentStorage::shred() const {
md[it.fieldName()] = valueElem.val.shred();
}
}
md.setMetadata(DocumentMetadataFields(metadata()));
return md.freeze();
}

View File

@ -450,6 +450,17 @@ TEST(ShredDocument, HandlesModifiedDocuments) {
ASSERT(!shredded["subObj"]["b"].missing());
}
TEST(ShredDocument, HandlesMetadata) {
BSONObj bson = BSON("a" << 1 << "subObj" << BSON("a" << 1));
Document original = fromBson(bson);
MutableDocument md(original);
DocumentMetadataFields meta;
meta.setSearchScore(6);
md.setMetadata(std::move(meta));
Document shredded = md.freeze().shred();
ASSERT_EQ(6, shredded.metadata().getSearchScore());
}
/** Add Document fields. */
class AddField {
public:

View File

@ -67,9 +67,10 @@ boost::intrusive_ptr<Expression> substituteInExpr(boost::intrusive_ptr<Expressio
/**
* Returns a vector of top-level dependencies where each index i in the vector corresponds to the
* dependencies from the ith expression according to 'orderToProcess'.
* dependencies from the ith expression according to 'orderToProcess'. Will return boost::none if
* any expression needs the whole document.
*/
std::vector<OrderedPathSet> getTopLevelDeps(
boost::optional<std::vector<OrderedPathSet>> getTopLevelDeps(
const std::vector<std::string>& orderToProcess,
const StringMap<boost::intrusive_ptr<Expression>>& expressions,
const StringMap<std::unique_ptr<ProjectionNode>>& children) {
@ -85,6 +86,10 @@ std::vector<OrderedPathSet> getTopLevelDeps(
childIt->second->reportDependencies(&deps);
}
if (deps.needWholeDocument) {
return boost::none;
}
topLevelDeps.push_back(
DepsTracker::simplifyDependencies(deps.fields, DepsTracker::TruncateToRootLevel::yes));
}
@ -117,9 +122,15 @@ std::pair<BSONObj, bool> InclusionNode::extractComputedProjectionsInProject(
return {BSONObj{}, false};
}
std::vector<OrderedPathSet> topLevelDeps =
boost::optional<std::vector<OrderedPathSet>> topLevelDeps =
getTopLevelDeps(_orderToProcessAdditionsAndChildren, _expressions, _children);
// If one of the expression requires the whole document, then we should not extract the
// projection and topLevelDeps will not hold any field names.
if (!topLevelDeps) {
return {BSONObj{}, false};
}
// Auxiliary vector with extracted computed projections: <name, expression, replacement
// strategy>. If the replacement strategy flag is true, the expression is replaced with a
// projected field. If it is false - the expression is replaced with an identity projection.
@ -147,12 +158,12 @@ std::pair<BSONObj, bool> InclusionNode::extractComputedProjectionsInProject(
// same projection depend on. If the extracted $addFields were to be placed before this
// projection, the dependency with the common name would be shadowed by the computed
// projection.
if (computedExprDependsOnField(topLevelDeps, field, i)) {
if (computedExprDependsOnField(topLevelDeps.get(), field, i)) {
replaceWithProjField = false;
continue;
}
const auto& topLevelFieldNames = topLevelDeps[i];
const auto& topLevelFieldNames = topLevelDeps.get()[i];
if (topLevelFieldNames.size() == 1 && topLevelFieldNames.count(oldName.toString()) == 1) {
// Substitute newName for oldName in the expression.
StringMap<std::string> renames;
@ -206,9 +217,15 @@ std::pair<BSONObj, bool> InclusionNode::extractComputedProjectionsInAddFields(
return {BSONObj{}, false};
}
std::vector<OrderedPathSet> topLevelDeps =
boost::optional<std::vector<OrderedPathSet>> topLevelDeps =
getTopLevelDeps(_orderToProcessAdditionsAndChildren, _expressions, _children);
// If one of the expression requires the whole document, then we should not extract the
// projection and topLevelDeps will not hold any field names.
if (!topLevelDeps) {
return {BSONObj{}, false};
}
// Auxiliary vector with extracted computed projections: <name, expression>.
// To preserve the original fields order, only projections at the beginning of the
// _orderToProcessAdditionsAndChildren list can be extracted for pushdown.
@ -229,11 +246,11 @@ std::pair<BSONObj, bool> InclusionNode::extractComputedProjectionsInAddFields(
// same projection depend on. If the extracted $addFields were to be placed before this
// projection, the dependency with the common name would be shadowed by the computed
// projection.
if (computedExprDependsOnField(topLevelDeps, field, i)) {
if (computedExprDependsOnField(topLevelDeps.get(), field, i)) {
break;
}
auto& topLevelFieldNames = topLevelDeps[i];
auto& topLevelFieldNames = topLevelDeps.get()[i];
if (topLevelFieldNames.size() == 1 && topLevelFieldNames.count(oldName.toString()) == 1) {
// Substitute newName for oldName in the expression.
StringMap<std::string> renames;

View File

@ -139,7 +139,7 @@ StringListSet MakeObjSpec::buildFieldDict(std::vector<std::string> names,
for (size_t i = 0; i < names.size(); ++i) {
size_t pos = fieldDict.findPos(names[i]);
if (pos != StringListSet::npos) {
auto& action = actions[i];
auto& action = actions[pos];
if (isClosed ? action.isKeep() : action.isDrop()) {
displayOrderSet.emplace(pos);

View File

@ -540,6 +540,12 @@ PlanState ScanStage::getNext() {
_seekRecordId,
*_coll.getCollName());
}
// Indicate that the last recordId seen is null once EOF is hit.
if (_recordIdSlot) {
auto [tag, val] = sbe::value::makeCopyRecordId(RecordId());
_recordIdAccessor.reset(true, tag, val);
}
_priority.reset();
return trackPlanState(PlanState::IS_EOF);
}

View File

@ -185,6 +185,8 @@ void UnwindStage::close() {
trackClose();
_children[0]->close();
_index = 0;
_inArray = false;
}
std::unique_ptr<PlanStageStats> UnwindStage::getStats(bool includeDebugInfo) const {
@ -221,8 +223,8 @@ std::vector<DebugPrinter::Block> UnwindStage::debugPrint() const {
return ret;
}
void UnwindStage::doSaveState(bool fullSave) {
if (!fullSave) {
void UnwindStage::doSaveState(bool relinquishCursor) {
if (!relinquishCursor) {
return;
}
if (_outFieldOutputAccessor) {
@ -233,12 +235,19 @@ void UnwindStage::doSaveState(bool fullSave) {
}
}
void UnwindStage::doRestoreState(bool fullSave) {
if (!slotsAccessible()) {
void UnwindStage::doRestoreState(bool relinquishCursor) {
if (!_inArray) {
// If we were once in an array but no longer are, this saves us from doing a refresh() on
// obsolete slot contents.
return;
}
_inArrayAccessor.refresh();
if (relinquishCursor) {
// The child stage will have copied the in-flight contents of this slot because WiredTiger
// will free the memory owned by the cursor it points to, so on restore we must update the
// embedded array iterator to point to the new memory location.
_inArrayAccessor.refresh();
}
}
size_t UnwindStage::estimateCompileTimeSize() const {

View File

@ -209,7 +209,7 @@ private:
bool _owned{false};
TypeTags _tag{TypeTags::Nothing};
Value _val{0};
};
}; // class OwnedValueAccessor
/**
* An accessor for a slot which must hold an array-like type (e.g. 'TypeTags::Array' or
@ -246,15 +246,17 @@ public:
}
void refresh() {
auto [tag, val] = _input->getViewOfValue();
_enumerator.reset(tag, val, _currentIndex);
if (_input) {
auto [tag, val] = _input->getViewOfValue();
_enumerator.reset(tag, val, _currentIndex);
}
}
private:
size_t _currentIndex = 0;
SlotAccessor* _input{nullptr};
ArrayEnumerator _enumerator;
};
}; // class ArrayAccessor
/**
* This is a switched accessor - it holds a vector of accessors and operates on an accessor selected

View File

@ -300,7 +300,7 @@ void TimeseriesModifyStage::_checkRestrictionsOnUpdatingShardKeyAreNotViolated(
// retryable write or in a transaction.
if (_params.allowShardKeyUpdatesWithoutFullShardKeyInQuery &&
feature_flags::gFeatureFlagUpdateOneWithoutShardKey.isEnabled(
serverGlobalParams.featureCompatibility)) {
serverGlobalParams.featureCompatibility.acquireFCVSnapshot())) {
bool isInternalThreadOrClient = !cc().session() || cc().isInternalClient();
uassert(ErrorCodes::InvalidOptions,
"$_allowShardKeyUpdatesWithoutFullShardKeyInQuery is an internal parameter",
@ -313,7 +313,7 @@ void TimeseriesModifyStage::_checkRestrictionsOnUpdatingShardKeyAreNotViolated(
// wouldChangeOwningShard error thrown below. If this node is a replica set secondary node,
// we can skip validation.
if (!feature_flags::gFeatureFlagUpdateDocumentShardKeyUsingTransactionApi.isEnabled(
serverGlobalParams.featureCompatibility)) {
serverGlobalParams.featureCompatibility.acquireFCVSnapshot())) {
uassert(ErrorCodes::IllegalOperation,
"Must run update to shard key field in a multi-statement transaction or with "
"retryWrites: true.",
@ -338,7 +338,7 @@ void TimeseriesModifyStage::_checkRestrictionsOnUpdatingShardKeyAreNotViolated(
// wouldChangeOwningShard error thrown below. If this node is a replica set secondary node,
// we can skip validation.
if (!feature_flags::gFeatureFlagUpdateDocumentShardKeyUsingTransactionApi.isEnabled(
serverGlobalParams.featureCompatibility)) {
serverGlobalParams.featureCompatibility.acquireFCVSnapshot())) {
uassert(ErrorCodes::IllegalOperation,
"Must run update to shard key field in a multi-statement transaction or with "
"retryWrites: true.",
@ -599,12 +599,16 @@ TimeseriesModifyStage::_writeToTimeseriesBuckets(ScopeGuard<F>& bucketFreer,
// it in memory.
[&] { /* noop */ });
if (status == NEED_YIELD && isEOF()) {
if (status == NEED_YIELD && isEOF() && !opCtx()->lockState()->inAWriteUnitOfWork()) {
// If this stage is already exhausted it won't use its children stages anymore and therefore
// it's okay if we failed to restore them. Avoid requesting a yield to the plan executor.
// Restoring from yield could fail due to a sharding placement change. Throwing a
// StaleConfig error is undesirable after an "update one" operation has already performed a
// write because the router would retry.
//
// If this plan is part of a larger encompassing WUOW it would be illegal to skip returning
// NEED_YIELD, so we don't skip it. In this case, such as multi-doc transactions, this is
// okay as the PlanExecutor is not allowed to auto-yield.
status = PlanStage::NEED_TIME;
}

View File

@ -169,7 +169,7 @@ void TimeseriesUpsertStage::_performInsert(BSONObj newMeasurement) {
// mongos will be able to start an internal transaction to handle the
// wouldChangeOwningShard error thrown below.
if (!feature_flags::gFeatureFlagUpdateDocumentShardKeyUsingTransactionApi.isEnabled(
serverGlobalParams.featureCompatibility)) {
serverGlobalParams.featureCompatibility.acquireFCVSnapshot())) {
uassert(ErrorCodes::IllegalOperation,
"The upsert document could not be inserted onto the shard targeted "
"by the query, since its shard key belongs on a different shard. "

View File

@ -597,13 +597,18 @@ PlanStage::StageState UpdateStage::doWork(WorkingSetID* out) {
});
if (restoreStateRet != PlanStage::NEED_TIME) {
if (restoreStateRet == PlanStage::NEED_YIELD && stageIsEOF) {
if (restoreStateRet == PlanStage::NEED_YIELD && stageIsEOF &&
!opCtx()->lockState()->inAWriteUnitOfWork()) {
// If this stage is already exhausted it won't use its children stages anymore and
// therefore it's okay if we failed to restore them. Avoid requesting a yield to the
// plan executor. Restoring from yield could fail due to a sharding placement
// change. Throwing a StaleConfig error is undesirable after an "update one"
// operation has already performed a write because the router would retry. Unset
// _idReturning as we'll return the document in this stage iteration.
//
// If this plan is part of a larger encompassing WUOW it would be illegal to skip
// returning NEED_YIELD, so we don't skip it. In this case, such as multi-doc
// transactions, this is okay as the PlanExecutor is not allowed to auto-yield.
_idReturning = WorkingSet::INVALID_ID;
} else {
return restoreStateRet;
@ -698,7 +703,7 @@ void UpdateStage::_checkRestrictionsOnUpdatingShardKeyAreNotViolated(
// retryable write or in a transaction.
if (_params.request->getAllowShardKeyUpdatesWithoutFullShardKeyInQuery().has_value() &&
feature_flags::gFeatureFlagUpdateOneWithoutShardKey.isEnabled(
serverGlobalParams.featureCompatibility)) {
serverGlobalParams.featureCompatibility.acquireFCVSnapshot())) {
bool isInternalThreadOrClient = !cc().session() || cc().isInternalClient();
uassert(ErrorCodes::InvalidOptions,
"$_allowShardKeyUpdatesWithoutFullShardKeyInQuery is an internal parameter",
@ -711,7 +716,7 @@ void UpdateStage::_checkRestrictionsOnUpdatingShardKeyAreNotViolated(
// wouldChangeOwningShard error thrown below. If this node is a replica set secondary node,
// we can skip validation.
if (!feature_flags::gFeatureFlagUpdateDocumentShardKeyUsingTransactionApi.isEnabled(
serverGlobalParams.featureCompatibility)) {
serverGlobalParams.featureCompatibility.acquireFCVSnapshot())) {
uassert(ErrorCodes::IllegalOperation,
"Must run update to shard key field in a multi-statement transaction or with "
"retryWrites: true.",
@ -734,7 +739,7 @@ void UpdateStage::_checkRestrictionsOnUpdatingShardKeyAreNotViolated(
// wouldChangeOwningShard error thrown below. If this node is a replica set secondary node,
// we can skip validation.
if (!feature_flags::gFeatureFlagUpdateDocumentShardKeyUsingTransactionApi.isEnabled(
serverGlobalParams.featureCompatibility)) {
serverGlobalParams.featureCompatibility.acquireFCVSnapshot())) {
uassert(ErrorCodes::IllegalOperation,
"Must run update to shard key field in a multi-statement transaction or with "
"retryWrites: true.",

View File

@ -179,7 +179,7 @@ void UpsertStage::_performInsert(BSONObj newDocument) {
// will be able to start an internal transaction to handle the
// wouldChangeOwningShard error thrown below.
if (!feature_flags::gFeatureFlagUpdateDocumentShardKeyUsingTransactionApi.isEnabled(
serverGlobalParams.featureCompatibility)) {
serverGlobalParams.featureCompatibility.acquireFCVSnapshot())) {
uassert(
ErrorCodes::IllegalOperation,
"The upsert document could not be inserted onto the shard targeted by the "

View File

@ -54,20 +54,18 @@ public:
BSONObj generateSection(OperationContext* opCtx,
const BSONElement& configElement) const override {
BSONObjBuilder bob;
if (serverGlobalParams.featureCompatibility.isVersionInitialized()) {
bob.append(
"major",
multiversion::majorVersion(serverGlobalParams.featureCompatibility.getVersion()));
bob.append(
"minor",
multiversion::minorVersion(serverGlobalParams.featureCompatibility.getVersion()));
const ServerGlobalParams::FCVSnapshot fcvSnapshot =
serverGlobalParams.featureCompatibility.acquireFCVSnapshot();
if (fcvSnapshot.isVersionInitialized()) {
bob.append("major", multiversion::majorVersion(fcvSnapshot.getVersion()));
bob.append("minor", multiversion::minorVersion(fcvSnapshot.getVersion()));
int currentlyTransitioning = 0;
// (Generic FCV reference): append information to serverStatus on if we are in a state
// of transitioning to a new FCV (upgrading or downgrading).
if (serverGlobalParams.featureCompatibility.isUpgradingOrDowngrading()) {
const auto& [fromVersion, toVersion] = multiversion::getTransitionFCVFromAndTo(
serverGlobalParams.featureCompatibility.getVersion());
if (fcvSnapshot.isUpgradingOrDowngrading()) {
const auto& [fromVersion, toVersion] =
multiversion::getTransitionFCVFromAndTo(fcvSnapshot.getVersion());
currentlyTransitioning = 1;
// from is greater, we are downgrading
if (fromVersion > toVersion) {

View File

@ -67,7 +67,7 @@ FeatureFlag::FeatureFlag(bool enabled, StringData versionString, bool shouldBeFC
// If the functionality of this function changes, make sure that the isEnabled/isPresentAndEnabled
// functions in feature_flag_util.js also incorporate the change.
bool FeatureFlag::isEnabled(const ServerGlobalParams::FeatureCompatibility& fcv) const {
bool FeatureFlag::isEnabled(const ServerGlobalParams::FCVSnapshot fcv) const {
// If the feature flag is not FCV gated, return whether it is enabled.
if (!_shouldBeFCVGated) {
return _enabled;
@ -93,8 +93,8 @@ bool FeatureFlag::isEnabled(const ServerGlobalParams::FeatureCompatibility& fcv)
}
bool FeatureFlag::isEnabledUseLastLTSFCVWhenUninitialized(
const ServerGlobalParams::FeatureCompatibility& fcv) const {
if (serverGlobalParams.featureCompatibility.isVersionInitialized()) {
const ServerGlobalParams::FCVSnapshot fcv) const {
if (fcv.isVersionInitialized()) {
return isEnabled(fcv);
} else {
// (Generic FCV reference): This reference is needed for the feature flag check API.
@ -103,8 +103,8 @@ bool FeatureFlag::isEnabledUseLastLTSFCVWhenUninitialized(
}
bool FeatureFlag::isEnabledUseLatestFCVWhenUninitialized(
const ServerGlobalParams::FeatureCompatibility& fcv) const {
if (serverGlobalParams.featureCompatibility.isVersionInitialized()) {
const ServerGlobalParams::FCVSnapshot fcv) const {
if (fcv.isVersionInitialized()) {
return isEnabled(fcv);
} else {
// (Generic FCV reference): This reference is needed for the feature flag check API.

View File

@ -66,22 +66,20 @@ public:
* If the functionality of this function changes, make sure that the
* isEnabled/isPresentAndEnabled functions in feature_flag_util.js also incorporate the change.
*/
bool isEnabled(const ServerGlobalParams::FeatureCompatibility& fcv) const;
bool isEnabled(ServerGlobalParams::FCVSnapshot fcv) const;
/**
* Returns true if the flag is set to true and enabled for this FCV version. If the FCV version
* is unset, instead checks against the default last LTS FCV version.
*/
bool isEnabledUseLastLTSFCVWhenUninitialized(
const ServerGlobalParams::FeatureCompatibility& fcv) const;
bool isEnabledUseLastLTSFCVWhenUninitialized(ServerGlobalParams::FCVSnapshot fcv) const;
/**
* Returns true if the flag is set to true and enabled for this FCV version. If the FCV version
* is unset, instead checks against the latest FCV version.
*/
bool isEnabledUseLatestFCVWhenUninitialized(
const ServerGlobalParams::FeatureCompatibility& fcv) const;
bool isEnabledUseLatestFCVWhenUninitialized(ServerGlobalParams::FCVSnapshot fcv) const;
/**
* Returns true if this flag is enabled regardless of the current FCV version. When using this

View File

@ -1757,18 +1757,21 @@ std::pair<write_ops::UpdateCommandReply, BSONObj> FLEQueryInterfaceImpl::updateW
updateReply.getWriteCommandReplyBase().setRetriedStmtIds(
std::vector<std::int32_t>{reply.getRetriedStmtId().value()});
}
updateReply.getWriteCommandReplyBase().setN(reply.getLastErrorObject().getNumDocs());
if (reply.getLastErrorObject().getUpserted().has_value()) {
auto& lastErrorObject = reply.getLastErrorObject();
updateReply.getWriteCommandReplyBase().setN(lastErrorObject.getNumDocs());
if (lastErrorObject.getUpserted().has_value()) {
write_ops::Upserted upserted;
upserted.setIndex(0);
upserted.set_id(reply.getLastErrorObject().getUpserted().value());
upserted.set_id(lastErrorObject.getUpserted().value());
updateReply.setUpserted(std::vector<mongo::write_ops::Upserted>{upserted});
}
if (reply.getLastErrorObject().getNumDocs() > 0) {
updateReply.setNModified(1);
updateReply.getWriteCommandReplyBase().setN(1);
} else {
dassert(lastErrorObject.getUpdatedExisting().has_value());
if (lastErrorObject.getUpdatedExisting().value()) {
updateReply.setNModified(1);
}
}
}

View File

@ -593,7 +593,7 @@ IndexBuildsCoordinator::makeKillIndexBuildOnLowDiskSpaceAction() {
void act(OperationContext* opCtx, int64_t availableBytes) noexcept final {
if (!feature_flags::gIndexBuildGracefulErrorHandling.isEnabled(
serverGlobalParams.featureCompatibility)) {
serverGlobalParams.featureCompatibility.acquireFCVSnapshot())) {
LOGV2(6826200,
"Index build: disk space monitor detected we're low on storage space but "
"'featureFlagIndexBuildGracefulErrorHandling' is disabled. Ignoring it");
@ -2804,7 +2804,7 @@ void IndexBuildsCoordinator::_cleanUpTwoPhaseAfterNonShutdownFailure(
// (Ignore FCV check): This feature flag doesn't have any upgrade/downgrade concerns.
if (feature_flags::gIndexBuildGracefulErrorHandling.isEnabled(
serverGlobalParams.featureCompatibility) &&
serverGlobalParams.featureCompatibility.acquireFCVSnapshot()) &&
replState->canVoteForAbort()) {
// Always request an abort to the primary node, even if we are primary. If
// primary, the signal will loop back and cause an asynchronous external
@ -2945,7 +2945,7 @@ void IndexBuildsCoordinator::_runIndexBuildInner(
// to the primary node. Single-phase builds can also abort immediately, as the primary or
// standalone is the only node aware of the build.
if (!feature_flags::gIndexBuildGracefulErrorHandling.isEnabled(
serverGlobalParams.featureCompatibility)) {
serverGlobalParams.featureCompatibility.acquireFCVSnapshot())) {
// Index builds only check index constraints when committing. If an error occurs at that
// point, then the build is cleaned up while still holding the appropriate locks. The only
// errors that we cannot anticipate are user interrupts and shutdown errors.

View File

@ -108,7 +108,8 @@ void logProcessDetailsForLogRotate(ServiceContext* serviceContext) {
}
}
serverGlobalParams.featureCompatibility.logFCVWithContext("log rotation"_sd);
serverGlobalParams.featureCompatibility.acquireFCVSnapshot().logFCVWithContext(
"log rotation"_sd);
logProcessDetails(nullptr);
}

View File

@ -127,11 +127,11 @@ TEST(ExpressionOptimizeTest, IsValidText) {
// Valid: TEXT outside NOR.
ASSERT_OK(isValid("{$text: {$search: 's'}, $nor: [{a: 1}, {b: 1}]}", *findCommand));
// Invalid: TEXT inside NOR. Boolean expression simplifier does not simplify it.
// Invalid: TEXT inside NOR.
ASSERT_NOT_OK(isValid("{$nor: [{$text: {$search: 's'}}, {a: 1}]}", *findCommand));
// Valid: Boolean expression simplifier opens up $nor expressions.
ASSERT_OK(
// Invalid: TEXT inside NOR.
ASSERT_NOT_OK(
isValid("{$nor: ["
" {$or: ["
" {$text: {$search: 's'}},"

View File

@ -1719,14 +1719,10 @@ void shutdownTask(const ShutdownTaskArgs& shutdownArgs) {
analyze_shard_key::QueryAnalysisSampler::get(serviceContext).onShutdown();
}
// Shutdown the TransportLayer so that new connections aren't accepted
if (auto tl = serviceContext->getTransportLayerManager()) {
TimeElapsedBuilderScopedTimer scopedTimer(serviceContext->getFastClockSource(),
"Shut down the transport layer",
&shutdownTimeElapsedBuilder);
LOGV2_OPTIONS(
20562, {LogComponent::kNetwork}, "Shutdown: going to close listening sockets");
tl->shutdown();
// Inform the TransportLayers to stop accepting new connections.
if (auto tlm = serviceContext->getTransportLayerManager()) {
LOGV2_OPTIONS(8314100, {LogComponent::kNetwork}, "Shutdown: Closing listener sockets");
tlm->stopAcceptingSessions();
}
// Shut down the global dbclient pool so callers stop waiting for connections.
@ -1947,6 +1943,15 @@ void shutdownTask(const ShutdownTaskArgs& shutdownArgs) {
CatalogCacheLoader::get(serviceContext).shutDown();
}
// Finish shutting down the TransportLayers
if (auto tlm = serviceContext->getTransportLayerManager()) {
TimeElapsedBuilderScopedTimer scopedTimer(serviceContext->getFastClockSource(),
"Shut down the transport layer",
&shutdownTimeElapsedBuilder);
LOGV2_OPTIONS(20562, {LogComponent::kNetwork}, "Shutdown: Closing open transport sessions");
tlm->shutdown();
}
if (auto* healthLog = HealthLogInterface::get(serviceContext)) {
TimeElapsedBuilderScopedTimer scopedTimer(serviceContext->getFastClockSource(),
"Shut down the health log",

View File

@ -64,8 +64,7 @@ bool NamespaceString::isCollectionlessAggregateNS() const {
return coll() == collectionlessAggregateCursorCol;
}
bool NamespaceString::isLegalClientSystemNS(
const ServerGlobalParams::FeatureCompatibility& currentFCV) const {
bool NamespaceString::isLegalClientSystemNS() const {
auto collectionName = coll();
if (isAdminDB()) {
if (collectionName == "system.roles")

View File

@ -668,7 +668,7 @@ public:
* Returns true if a client can modify this namespace even though it is under ".system."
* For example <dbname>.system.users is ok for regular clients to update.
*/
bool isLegalClientSystemNS(const ServerGlobalParams::FeatureCompatibility& currentFCV) const;
bool isLegalClientSystemNS() const;
/**
* Returns true if this namespace refers to a drop-pending collection.

View File

@ -162,17 +162,15 @@ TEST_F(NamespaceStringTest, IsCollectionlessCursorNamespace) {
}
TEST_F(NamespaceStringTest, IsLegalClientSystemNamespace) {
const auto& currentFCV = serverGlobalParams.featureCompatibility;
ASSERT_TRUE(makeNamespaceString(boost::none, "test.system.buckets.1234")
.isLegalClientSystemNS(currentFCV));
ASSERT_TRUE(makeNamespaceString(boost::none, "test.system.buckets.abcde")
.isLegalClientSystemNS(currentFCV));
ASSERT_FALSE(makeNamespaceString(boost::none, "test.system.buckets..1234")
.isLegalClientSystemNS(currentFCV));
ASSERT_FALSE(makeNamespaceString(boost::none, "test.system.buckets.a234$")
.isLegalClientSystemNS(currentFCV));
ASSERT_TRUE(
makeNamespaceString(boost::none, "test.system.buckets.1234").isLegalClientSystemNS());
ASSERT_TRUE(
makeNamespaceString(boost::none, "test.system.buckets.abcde").isLegalClientSystemNS());
ASSERT_FALSE(
makeNamespaceString(boost::none, "test.system.buckets.").isLegalClientSystemNS(currentFCV));
makeNamespaceString(boost::none, "test.system.buckets..1234").isLegalClientSystemNS());
ASSERT_FALSE(
makeNamespaceString(boost::none, "test.system.buckets.a234$").isLegalClientSystemNS());
ASSERT_FALSE(makeNamespaceString(boost::none, "test.system.buckets.").isLegalClientSystemNS());
}
TEST_F(NamespaceStringTest, IsDropPendingNamespace) {

View File

@ -119,7 +119,7 @@ OpMsgFuzzerFixture::OpMsgFuzzerFixture(bool skipGlobalInitializers)
storageGlobalParams.repair = false;
serverGlobalParams.enableMajorityReadConcern = false;
// (Generic FCV reference): Initialize FCV.
serverGlobalParams.mutableFeatureCompatibility.setVersion(multiversion::GenericFCV::kLatest);
serverGlobalParams.mutableFCV.setVersion(multiversion::GenericFCV::kLatest);
initializeStorageEngine(opCtx.get(),
StorageEngineInitFlags::kAllowNoLockFile |

View File

@ -85,17 +85,19 @@ void FcvOpObserver::_setVersion(OperationContext* opCtx,
FeatureCompatibilityVersion::advanceLastFCVUpdateTimestamp(*commitTs);
boost::optional<multiversion::FeatureCompatibilityVersion> prevVersion;
if (serverGlobalParams.featureCompatibility.isVersionInitialized()) {
prevVersion = serverGlobalParams.featureCompatibility.getVersion();
const auto prevFcvSnapshot = serverGlobalParams.featureCompatibility.acquireFCVSnapshot();
if (prevFcvSnapshot.isVersionInitialized()) {
prevVersion = prevFcvSnapshot.getVersion();
}
serverGlobalParams.mutableFeatureCompatibility.setVersion(newVersion);
serverGlobalParams.featureCompatibility.logFCVWithContext("setFCV"_sd);
serverGlobalParams.mutableFCV.setVersion(newVersion);
const auto newFcvSnapshot = serverGlobalParams.featureCompatibility.acquireFCVSnapshot();
newFcvSnapshot.logFCVWithContext("setFCV"_sd);
FeatureCompatibilityVersion::updateMinWireVersion(opCtx);
// (Generic FCV reference): This FCV check should exist across LTS binary versions.
if (serverGlobalParams.featureCompatibility.isGreaterThanOrEqualTo(
multiversion::GenericFCV::kLatest) ||
serverGlobalParams.featureCompatibility.isUpgradingOrDowngrading()) {
if (newFcvSnapshot.isGreaterThanOrEqualTo(multiversion::GenericFCV::kLatest) ||
newFcvSnapshot.isUpgradingOrDowngrading()) {
// minWireVersion == maxWireVersion on kLatest FCV or upgrading/downgrading FCV.
// Close all incoming connections from internal clients with binary versions lower than
// ours.
@ -121,7 +123,7 @@ void FcvOpObserver::_setVersion(OperationContext* opCtx,
// in the upgrading/downgrading state.
// (Generic FCV reference): This FCV check should exist across LTS binary versions.
try {
if (serverGlobalParams.featureCompatibility.isUpgradingOrDowngrading()) {
if (newFcvSnapshot.isUpgradingOrDowngrading()) {
SessionKiller::Matcher matcherAllSessions(
KillAllSessionsByPatternSet{makeKillAllSessionsByPattern(opCtx)});
killSessionsAbortUnpreparedTransactions(opCtx, matcherAllSessions);
@ -177,8 +179,9 @@ void FcvOpObserver::_onInsertOrUpdate(OperationContext* opCtx, const BSONObj& do
// version changes.
logv2::DynamicAttributes attrs;
bool isDifferent = true;
if (serverGlobalParams.featureCompatibility.isVersionInitialized()) {
const auto currentVersion = serverGlobalParams.featureCompatibility.getVersion();
const auto fcvSnapshot = serverGlobalParams.featureCompatibility.acquireFCVSnapshot();
if (fcvSnapshot.isVersionInitialized()) {
const auto currentVersion = fcvSnapshot.getVersion();
attrs.add("currentVersion", multiversion::toString(currentVersion));
isDifferent = currentVersion != newVersion;
}
@ -247,7 +250,8 @@ void FcvOpObserver::onReplicationRollback(OperationContext* opCtx,
if (swFcv.isOK()) {
const auto featureCompatibilityVersion = swFcv.getValue();
auto swVersion = FeatureCompatibilityVersionParser::parse(featureCompatibilityVersion);
const auto memoryFcv = serverGlobalParams.featureCompatibility.getVersion();
const auto memoryFcv =
serverGlobalParams.featureCompatibility.acquireFCVSnapshot().getVersion();
if (swVersion.isOK() && (swVersion.getValue() != memoryFcv)) {
auto diskFcv = swVersion.getValue();
LOGV2(4675801,

View File

@ -1689,7 +1689,8 @@ void OpObserverImpl::onBatchedWriteCommit(OperationContext* opCtx) {
getMaxSizeOfBatchedOperationsInSingleOplogEntryBytes(),
/*prepare=*/false);
if (!gFeatureFlagLargeBatchedOperations.isEnabled(serverGlobalParams.featureCompatibility)) {
if (!gFeatureFlagLargeBatchedOperations.isEnabled(
serverGlobalParams.featureCompatibility.acquireFCVSnapshot())) {
// Before SERVER-70765, we relied on packTransactionStatementsForApplyOps() to check if the
// batch of operations could fit in a single applyOps entry. Now, we pass the size limit to
// TransactionOperations::getApplyOpsInfo() and are now able to return an error earlier.

View File

@ -225,7 +225,7 @@ Status userAllowedCreateNS(OperationContext* opCtx, const NamespaceString& ns) {
return Status::OK();
}
if (ns.isSystem() && !ns.isLegalClientSystemNS(serverGlobalParams.featureCompatibility)) {
if (ns.isSystem() && !ns.isLegalClientSystemNS()) {
return Status(ErrorCodes::InvalidNamespace,
str::stream() << "Invalid system namespace: " << ns.toStringForErrorMsg());
}
@ -248,7 +248,7 @@ Status userAllowedCreateNS(OperationContext* opCtx, const NamespaceString& ns) {
return Status::OK();
}
if (ns.isConfigDB() && ns.isLegalClientSystemNS(serverGlobalParams.featureCompatibility)) {
if (ns.isConfigDB() && ns.isLegalClientSystemNS()) {
return Status::OK();
}

View File

@ -62,12 +62,13 @@ ParsedDelete::ParsedDelete(OperationContext* opCtx,
: _opCtx(opCtx),
_request(request),
_collection(collection),
_timeseriesDeleteQueryExprs(isTimeseriesDelete
? createTimeseriesWritesQueryExprsIfNecessary(
feature_flags::gTimeseriesDeletesSupport.isEnabled(
serverGlobalParams.featureCompatibility),
collection)
: nullptr),
_timeseriesDeleteQueryExprs(
isTimeseriesDelete
? createTimeseriesWritesQueryExprsIfNecessary(
feature_flags::gTimeseriesDeletesSupport.isEnabled(
serverGlobalParams.featureCompatibility.acquireFCVSnapshot()),
collection)
: nullptr),
_isRequestToTimeseries(isTimeseriesDelete) {}
Status ParsedDelete::parseRequest() {

View File

@ -102,12 +102,13 @@ ParsedUpdateBase::ParsedUpdateBase(OperationContext* opCtx,
_canonicalQuery(),
_extensionsCallback(std::move(extensionsCallback)),
_collection(collection),
_timeseriesUpdateQueryExprs(isRequestToTimeseries
? createTimeseriesWritesQueryExprsIfNecessary(
feature_flags::gTimeseriesUpdatesSupport.isEnabled(
serverGlobalParams.featureCompatibility),
collection)
: nullptr),
_timeseriesUpdateQueryExprs(
isRequestToTimeseries
? createTimeseriesWritesQueryExprsIfNecessary(
feature_flags::gTimeseriesUpdatesSupport.isEnabled(
serverGlobalParams.featureCompatibility.acquireFCVSnapshot()),
collection)
: nullptr),
_isRequestToTimeseries(isRequestToTimeseries) {
if (forgoOpCounterIncrements) {
_expCtx->enabledCounters = false;

View File

@ -2263,7 +2263,7 @@ void tryPerformTimeseriesBucketCompression(
const write_ops::InsertCommandRequest& request) {
// When enabled, we skip constructing ClosedBuckets which results in skipping compression.
invariant(!feature_flags::gTimeseriesAlwaysUseCompressedBuckets.isEnabled(
serverGlobalParams.featureCompatibility));
serverGlobalParams.featureCompatibility.acquireFCVSnapshot()));
// Buckets with just a single measurement is not worth compressing.
if (closedBucket.numMeasurements.has_value() && closedBucket.numMeasurements.value() <= 1) {

View File

@ -229,7 +229,7 @@ void validate(OperationContext* opCtx,
"$_requestResumeToken is not supported without Resharding Improvements",
!requestResumeTokenElem ||
resharding::gFeatureFlagReshardingImprovements.isEnabled(
serverGlobalParams.featureCompatibility));
serverGlobalParams.featureCompatibility.acquireFCVSnapshot()));
uassert(ErrorCodes::FailedToParse,
str::stream() << AggregateCommandRequest::kRequestResumeTokenFieldName
<< " must be a boolean type",

View File

@ -331,7 +331,7 @@ DocumentSourceChangeStreamUnwindTransaction::TransactionOpIterator::TransactionO
// We need endOfTransaction only for unprepared transactions: so this must be an applyOps with
// set lsid and txnNumber.
_needEndOfTransaction = feature_flags::gFeatureFlagEndOfTransactionChangeEvent.isEnabled(
serverGlobalParams.featureCompatibility) &&
serverGlobalParams.featureCompatibility.acquireFCVSnapshot()) &&
!applyOps.missing() && _lsid.has_value() && _txnNumber.has_value();
if (BSONType::Object ==

View File

@ -126,8 +126,8 @@ TEST_F(InternalUnpackBucketPushdownProjectionsTest, OptimizeAddFieldsWith2MetaPr
TEST_F(InternalUnpackBucketPushdownProjectionsTest, SplitAddFieldsWithMixedProjectionFields) {
auto unpackSpecObj = fromjson(
"{$_internalUnpackBucket: { exclude: [], timeField: 'foo', metaField: 'myMeta', "
"bucketMaxSpanSeconds: 3600}}");
"{$_internalUnpackBucket: { exclude: [], timeField: 'foo', metaField: "
"'myMeta',bucketMaxSpanSeconds: 3600}}");
auto addFieldsSpecObj =
fromjson("{$addFields: {device: '$myMeta.a', temp: {$add: ['$temperature', '$offset']}}}");
@ -349,5 +349,110 @@ TEST_F(InternalUnpackBucketPushdownProjectionsTest, DoNotPushDownNestedProjectio
ASSERT_BSONOBJ_EQ(projectSpecObj, serialized[1]);
}
/****************** $project stage with $getField expression ****************************/
// We do not push down projections with the '$getField' expression when the input to '$getField' is
// just a string. In this case $getField will always prepend the $$CURRENT field path for string
// inputs and thus also require the 'needWholeDocument' dependency. So for all values of {$getField:
// "string"} we will not perform this rewrite. Even though, we could perform the rewrite here when
// the string is the metaField, the server cannot differentiate between 'meta' and '$meta' field
// paths, where one is the metaField and the other is not in the expression dependencies. To avoid
// incorrect query results in this edge case, we restrict all rewrites with {$getField: "string"}.
// Note that we do not expect users to use $getField to query their metaField.
TEST_F(InternalUnpackBucketPushdownProjectionsTest,
DoNotPushDownNestedProjectionWithGetFieldJustString) {
auto unpackSpecObj = fromjson(
"{$_internalUnpackBucket: { exclude: [], timeField: 'time', metaField: 'myMeta', "
"bucketMaxSpanSeconds: 3600}}");
auto projectSpecObj = fromjson(
"{$project: {_id: true, x: true, data: {z: {$add: [{$getField: 'myMeta'}, "
"'$myMeta.b']}}}}");
auto pipeline = Pipeline::parse(makeVector(unpackSpecObj, projectSpecObj), getExpCtx());
auto& container = pipeline->getSources();
auto unpack = dynamic_cast<DocumentSourceInternalUnpackBucket*>(container.begin()->get());
auto nextStageIsRemoved = unpack->pushDownComputedMetaProjection(container.begin(), &container);
ASSERT_EQ(nextStageIsRemoved, false);
auto serialized = pipeline->serializeToBson();
ASSERT_EQ(2u, serialized.size());
auto projectSerialized = fromjson(
"{$project: {_id: true, x: true, data: {z: {$add: [{$getField: { field: { $const: "
"'myMeta'}, input:'$$CURRENT' } },'$myMeta.b']}}}}");
ASSERT_BSONOBJ_EQ(projectSerialized, serialized[1]);
}
// However, we can push down $getField if we have the entire path and do not rely on $$CURRENT. If
// the entire path is only on the metaField, we can pushdown the projection.
TEST_F(InternalUnpackBucketPushdownProjectionsTest, DoPushDownNestedProjectionWithGetFieldInput) {
auto unpackSpecObj = fromjson(
"{$_internalUnpackBucket: { exclude: [], timeField: 'time', metaField: 'myMeta', "
"bucketMaxSpanSeconds: 3600}}");
auto projectSpecObj =
fromjson("{$project: {_id : true, device: {$getField: {input: '$myMeta', field:'a'}}}}");
auto pipeline = Pipeline::parse(makeVector(unpackSpecObj, projectSpecObj), getExpCtx());
auto& container = pipeline->getSources();
auto unpack = dynamic_cast<DocumentSourceInternalUnpackBucket*>(container.begin()->get());
auto nextStageIsRemoved = unpack->pushDownComputedMetaProjection(container.begin(), &container);
ASSERT_EQ(nextStageIsRemoved, false);
auto serialized = pipeline->serializeToBson();
ASSERT_EQ(3u, serialized.size());
ASSERT_BSONOBJ_EQ(
fromjson(
"{$addFields: { device: { $getField: { field: { $const: 'a' }, input: '$meta' }}}}"),
serialized[0]);
ASSERT_BSONOBJ_EQ(fromjson("{$_internalUnpackBucket: { exclude: [], timeField: 'time', "
"metaField: 'myMeta', bucketMaxSpanSeconds: 3600, "
"computedMetaProjFields: ['device']}}"),
serialized[1]);
ASSERT_BSONOBJ_EQ(fromjson("{$project: {_id : true, device: true}}"), serialized[2]);
}
TEST_F(InternalUnpackBucketPushdownProjectionsTest,
DoNotPushDownNestedProjectionWithMeasurementGetField) {
auto unpackSpecObj = fromjson(
"{$_internalUnpackBucket: { exclude: [], timeField: 'time', metaField: 'myMeta', "
"bucketMaxSpanSeconds: 3600}}");
auto projectSpecObj = fromjson(
"{$project: {_id: true, x: true, data: {z: {$add: [{$getField: {input: '$other', "
"field:'a'}},'$myMeta.b']}}}}");
auto pipeline = Pipeline::parse(makeVector(unpackSpecObj, projectSpecObj), getExpCtx());
auto& container = pipeline->getSources();
auto unpack = dynamic_cast<DocumentSourceInternalUnpackBucket*>(container.begin()->get());
auto nextStageIsRemoved = unpack->pushDownComputedMetaProjection(container.begin(), &container);
ASSERT_EQ(nextStageIsRemoved, false);
auto serialized = pipeline->serializeToBson();
ASSERT_EQ(2u, serialized.size());
auto projectSerialized = fromjson(
"{$project: {_id: true, x: true, data: {z: {$add: [{$getField: { field: {$const: 'a'}, "
"input:'$other' } },'$myMeta.b']}}}}");
ASSERT_BSONOBJ_EQ(projectSerialized, serialized[1]);
}
TEST_F(InternalUnpackBucketPushdownProjectionsTest,
DoNotPushDownNestedProjectionWhichNeedsWholeDoc) {
auto unpackSpecObj = fromjson(
"{$_internalUnpackBucket: { exclude: [], timeField: 'time', metaField: 'myMeta', "
"bucketMaxSpanSeconds: 3600}}");
auto projectSpecObj = fromjson("{$project: {_id: true, x: true, data: '$$ROOT'}}");
auto pipeline = Pipeline::parse(makeVector(unpackSpecObj, projectSpecObj), getExpCtx());
auto& container = pipeline->getSources();
auto unpack = dynamic_cast<DocumentSourceInternalUnpackBucket*>(container.begin()->get());
auto nextStageIsRemoved = unpack->pushDownComputedMetaProjection(container.begin(), &container);
ASSERT_EQ(nextStageIsRemoved, false);
auto serialized = pipeline->serializeToBson();
ASSERT_EQ(2u, serialized.size());
ASSERT_BSONOBJ_EQ(unpackSpecObj, serialized[0]);
ASSERT_BSONOBJ_EQ(projectSpecObj, serialized[1]);
}
} // namespace
} // namespace mongo

View File

@ -127,7 +127,7 @@ intrusive_ptr<DocumentSource> DocumentSourceListCatalog::createFromBson(
uassert(ErrorCodes::QueryFeatureNotAllowed,
fmt::format("The {} aggregation stage is not enabled", kStageName),
feature_flags::gDocumentSourceListCatalog.isEnabled(
serverGlobalParams.featureCompatibility));
serverGlobalParams.featureCompatibility.acquireFCVSnapshot()));
return new DocumentSourceListCatalog(pExpCtx);
}

View File

@ -190,7 +190,7 @@ void DocumentSourceOut::initialize() {
uassert(7406100,
"$out to time-series collections is only supported on FCV greater than or equal to 7.1",
feature_flags::gFeatureFlagAggOutTimeseries.isEnabled(
serverGlobalParams.featureCompatibility) ||
serverGlobalParams.featureCompatibility.acquireFCVSnapshot()) ||
!_timeseries);
const NamespaceString& outputNs = makeBucketNsIfTimeseries(getOutputNs());
@ -270,7 +270,7 @@ void DocumentSourceOut::finalize() {
uassert(7406101,
"$out to time-series collections is only supported on FCV greater than or equal to 7.1",
feature_flags::gFeatureFlagAggOutTimeseries.isEnabled(
serverGlobalParams.featureCompatibility) ||
serverGlobalParams.featureCompatibility.acquireFCVSnapshot()) ||
!_timeseries);
// If the collection is time-series, we must rename to the "real" buckets collection.

View File

@ -39,7 +39,6 @@
#include "mongo/base/error_codes.h"
#include "mongo/bson/bsonobj.h"
#include "mongo/bson/bsontypes.h"
#include "mongo/bson/timestamp.h"
#include "mongo/db/commands/server_status_metric.h"
#include "mongo/db/feature_flag.h"
#include "mongo/db/pipeline/document_source_query_stats_gen.h"
@ -239,7 +238,7 @@ DocumentSource::GetNextResult DocumentSourceQueryStats::doGetNext() {
}
boost::optional<Document> DocumentSourceQueryStats::toDocument(
const Timestamp& partitionReadTime, const QueryStatsEntry& queryStatsEntry) const {
const Date_t& partitionReadTime, const QueryStatsEntry& queryStatsEntry) const {
const auto& key = queryStatsEntry.key;
const auto& hash = absl::HashOf(key);
try {
@ -287,7 +286,7 @@ void DocumentSourceQueryStats::CopiedPartition::load(QueryStatsStore& queryStats
statsEntries.clear();
// Capture the time at which reading the partition begins.
_readTimestamp = Timestamp(Date_t::now().toMillisSinceEpoch() / 1000, 0);
_readTimestamp = Date_t::now();
{
// We only keep the partition (which holds a lock)
// for the time needed to collect the metrics (QueryStatsEntry)
@ -317,7 +316,7 @@ bool DocumentSourceQueryStats::CopiedPartition::isValidPartitionId(
return _partitionId < maxNumPartitions;
}
const Timestamp& DocumentSourceQueryStats::CopiedPartition::getReadTimestamp() const {
const Date_t& DocumentSourceQueryStats::CopiedPartition::getReadTimestamp() const {
return _readTimestamp;
}

View File

@ -170,7 +170,7 @@ private:
bool isValidPartitionId(QueryStatsStore::PartitionId maxNumPartitions) const;
const Timestamp& getReadTimestamp() const;
const Date_t& getReadTimestamp() const;
bool empty() const;
@ -179,7 +179,7 @@ private:
std::deque<QueryStatsEntry> statsEntries;
private:
Timestamp _readTimestamp;
Date_t _readTimestamp;
QueryStatsStore::PartitionId _partitionId;
bool _isLoaded{false};
};
@ -198,7 +198,7 @@ private:
GetNextResult doGetNext() final;
boost::optional<Document> toDocument(const Timestamp& partitionReadTime,
boost::optional<Document> toDocument(const Date_t& partitionReadTime,
const QueryStatsEntry& queryStatsEntry) const;
// The current partition copied from query stats store to avoid holding lock during reads.

View File

@ -519,7 +519,8 @@ std::vector<std::unique_ptr<InnerPipelineStageInterface>> findSbeCompatibleStage
// TODO (SERVER-80243): Remove 'featureFlagTimeSeriesInSbe' check.
.unpackBucket = feature_flags::gFeatureFlagTimeSeriesInSbe.isEnabled(
serverGlobalParams.featureCompatibility) &&
serverGlobalParams.featureCompatibility.acquireFCVSnapshot()) &&
!queryKnob.getSbeDisableTimeSeriesForOp() &&
cq->getExpCtx()->sbePipelineCompatibility == SbeCompatibility::fullyCompatible,
};

Some files were not shown because too many files have changed in this diff Show More