Compare commits

..

332 Commits
master ... v2.9

Author SHA1 Message Date
Shane Harvey
4828e426d8 BUMP 2.9.5 2017-06-30 15:13:06 -07:00
Shane Harvey
bc868e0dc6 PYTHON-1293 Evergreen - disable Solaris testing. 2017-06-30 15:03:11 -07:00
Tzach
ccc1d2037d Slightly faster Cursor.next() (#323) 2017-06-29 12:53:54 -07:00
Shane Harvey
e3c58c0325 PYTHON-1291 Skip *_with_invalid_keys tests against >= 3.5.8. 2017-06-28 18:05:21 -07:00
Bernie Hackett
b067e804f0 PYTHON-1285 - Switch Evergreen OSX distro to 1012 2017-06-22 16:54:04 -07:00
Bernie Hackett
3d49b2f5f6 Fix test_comment for MongoDB 3.5 2017-05-25 16:59:48 -07:00
Bernie Hackett
c3dc1f4a8b Try to fix Travis again 2017-05-25 14:17:36 -07:00
Bernie Hackett
61f9504230 Attempt to fix Travis for python 3.2 / pypy3-2.4 2017-05-25 13:39:42 -07:00
Bernie Hackett
a7402900b8 Document 2.x branch status on index page 2017-05-25 13:22:56 -07:00
Bernie Hackett
b090eca6cc Github doesn't support the warning directive 2017-05-25 13:17:47 -07:00
Bernie Hackett
a29a4f7da1 Document 2.x branch status in the README 2017-05-25 13:14:46 -07:00
Bernie Hackett
01f773c9f7 Test mod_wsgi in Evergreen 2017-04-27 12:14:02 -07:00
Bernie Hackett
3d9e2eab22 Update Travis config 2017-04-26 22:24:13 -07:00
Bernie Hackett
099453fd81 Migrate testing to Ubuntu 12.04
This commit also cleans up the Evergreen config
file and fixes a test for MongoDB 3.5+.
2017-04-26 21:13:43 -07:00
Bernie Hackett
b95a4d8457 PYTHON-1201 - Test GSSAPI and PLAIN authentication 2017-03-14 13:05:17 -07:00
Bernie Hackett
d2a5cd82b0 Changelog updates 2017-03-06 16:18:35 -08:00
Bernie Hackett
7f73d3764e Various fixes for Sphinx doc builds 2017-03-06 15:35:27 -08:00
Bernie Hackett
d5ccdb48b0 Fix UUID tests with the pypi uuid module
This change skips tests of CSHARP_LEGACY when the third
party uuid module from pypi is installed (or any other
uuid module that doesn't support bytes_le).
2017-03-06 12:40:54 -08:00
Shane Harvey
c6b1c2854e Evergreen - enable Windows SSL tests 2017-03-03 13:36:18 -08:00
Bernie Hackett
ffeebbc486 PYTHON-1247 - Work around distutils MingW32 issues 2017-03-03 12:02:37 -08:00
Bernie Hackett
96891e921c PYTHON-1245 - Fix MinGW32 build warning 2017-02-27 17:32:47 -08:00
Bernie Hackett
f2b9ec844f PYTHON-1200 - Test CPython 2.6, 2.7, and 3.2 on Windows 2017-02-27 09:58:20 -08:00
Shane Harvey
7702c46957 PYTHON-1232 Add Jenkins storage engines test matrix 2017-02-23 15:10:30 -08:00
Bernie Hackett
959b395da0 Fix infrequent read preference test failure 2017-02-17 16:02:06 -08:00
Bernie Hackett
c4091d8d75 PYTHON-1239 - Test explicitly without C extensions 2017-02-17 13:41:13 -08:00
Bernie Hackett
a87b390e82 Test C extensions on Windows - Python 3.3 - 3.6 2017-02-09 18:22:15 -08:00
Bernie Hackett
19e1161e86 Keep Jython 2.5 tests from running out of memory 2017-02-08 15:56:00 -08:00
Bernie Hackett
c27f382a17 Disable evergreen "upload working dir" task
It's unnecessary and causes CI timeouts.
2017-02-08 14:04:59 -08:00
Bernie Hackett
39cf311c74 PYTHON-1158 - Protect pool init and fix auth test 2017-02-07 15:26:33 -08:00
Bernie Hackett
8e57445794 PYTHON-1202 - Fix time64 compilation with -std=c99
Python 2.6 on Solaris 11 is built with -std=c99. That causes
build issues for our time64 code, which needs localtime_r and
tzset. Including Python.h in time64.c provides the necessary magic.
2017-02-07 14:36:44 -08:00
Shane Harvey
45fdd780fc Evergreen: add Python 3.2 testing 2017-02-07 12:22:44 -08:00
Bernie Hackett
467ff3f8ee PYTHON-1238 - Don't send empty writeConcern to user management commands 2017-02-07 11:41:19 -08:00
Bernie Hackett
9e123e3c11 PYTHON-1235 - Fix auto reconnect test under Jython 2.7 2017-02-06 13:38:11 -08:00
Bernie Hackett
80fdf61cdb PYTHON-1205 - Use SIGALRM instead of interrupt_main on non-Windows 2017-02-06 13:36:23 -08:00
Bernie Hackett
544fd06f6f Fix tests for gevent 1.2
Gevent 1.2 replaced gevent.coros with gevent.lock.
2017-02-06 13:33:54 -08:00
Bernie Hackett
090a39be10 Improve master slave test setup 2017-02-06 13:32:36 -08:00
Bernie Hackett
7ff4898e97 Improve test suite primary discovery 2017-02-06 13:30:49 -08:00
Bernie Hackett
295bd96648 PYTHON-1237 - Update aggregate test for MongoDB 3.5+ 2017-02-06 13:24:14 -08:00
Bernie Hackett
0308797cca Update TLS test certificates and test configuration 2017-02-06 13:23:00 -08:00
Shane Harvey
489ef3676e PYTHON-1198 Test PyMongo v2.9 on Evergreen.
Install nose via setup_requires to produce XML test output.
Add Python 2.4, 2.5, Jython 2.5 testing.
Only run the SSL test suite when SSL is enabled.
test_ssl.py should not fail when auth_context.client does not exist.
Work around the nosetests command and verbosity flag on Python 3.
2017-02-03 15:54:46 -08:00
Shane Harvey
d5a7120e6a Add Evergreen config from evergreen branch @287a87eee9ef4a2a0ec4a60fcb8f785ca1af52d3 2017-02-02 16:58:38 -08:00
Bernie Hackett
53ee18eefe Changelog updates for 2.9.5 2017-01-31 13:36:14 -08:00
Bernie Hackett
a28c4f1d4e Claim support for Python 3.6 2017-01-31 13:25:00 -08:00
Bernie Hackett
4f52cd3f5d PYTHON-1158 - Use SSLContext and PROTOCOL_TLS_CLIENT
This change works around deprecations in CPython 3.6 and expected
deprecations in later Python releases.
2017-01-31 13:08:04 -08:00
Bernie Hackett
5d2195d865 Start work on 2.9.5 2017-01-31 13:06:25 -08:00
Bernie Hackett
1c136a627b BUMP 2.9.4 2016-09-30 10:46:45 -07:00
Bernie Hackett
7903473f99 Changelog for 2.9.4 2016-09-29 17:19:02 -07:00
Bernie Hackett
989a40ade5 PYTHON-1144 - Add Atlas / PyMongo 2.x usage guide 2016-09-29 15:04:18 -07:00
Bernie Hackett
500db80aa0 PYTHON-1154 - Various doc and doctest fixes
- Make the doctests pass against MongoDB 2.6 - master (3.3)
- Make the doctests pass under python 2.6 - python 3.5
- Make the docs build properly with python 2.4 and 2.5

Previous changes to make the doctests pass under python 3 keep
them from working at all under python 2.4, and cause some doctests
to fail under python 2.5. That feels like a fair tradeoff.
2016-09-28 15:57:34 -07:00
Bernie Hackett
2c232e78b2 PYTHON-1153 - Make docs build and test under python 3 2016-09-26 15:39:57 -07:00
Bernie Hackett
f37be740e2 Remove test assumptions about primary host 2016-09-21 17:31:19 -07:00
A. Jesse Jiryu Davis
c7c352ff3d PYTHON-1145 RS client obeys "uuidRepresentation=". 2016-09-19 17:19:06 -04:00
Bernie Hackett
d172aeb542 PYTHON-1092 - Backport Cursor.address from PyMongo 3.x 2016-05-19 17:30:47 -07:00
Bernie Hackett
de847f03b0 PYTHON-1094 - Document how to use TLS securely in 2.x 2016-05-19 14:32:25 -07:00
Bernie Hackett
753ef14b7f PYTHON-1088 - Fix various MongoClient __repr__ issues 2016-05-19 14:16:35 -07:00
Bernie Hackett
478726267b Start work on 2.9.4 2016-05-18 16:11:58 -07:00
Luke Lovett
f185c87b47 BUMP 2.9.3 2016-03-15 14:26:19 -07:00
Bernie Hackett
f6e2adbb45 Changelog for 2.9.3 2016-03-14 17:01:53 -07:00
Bernie Hackett
6f3e23b9f4 Test debugging 2016-03-10 14:36:17 -08:00
Bernie Hackett
64801b45c6 Fix a racy test 2016-03-10 12:50:35 -08:00
Bernie Hackett
264cdd8b5a PYTHON-1070 - Make index cache thread safe 2016-03-10 10:12:02 -08:00
Bernie Hackett
cb4a80a28a PYTHON-1063 - Add an example doc for tailable cursors 2016-03-08 18:06:18 -08:00
Bernie Hackett
57cc383286 Fix a test for MongoDB 3.3 behavior change 2016-03-03 16:17:50 -05:00
Bernie Hackett
4d9831f97c PYTHON-1060 - Remove command document repr from exception messages 2016-03-02 17:05:37 -05:00
Bernie Hackett
27a232cd40 PYTHON-1056 - Disallow double quotes in database names 2016-03-02 16:38:02 -05:00
Bernie Hackett
cd66d35213 PYTHON-1062 - Only use "textSearchEnabled" when testing MongoDB 2.4 2016-03-02 15:08:58 -05:00
Bernie Hackett
43525029bf Start work on 2.9.3 2016-03-02 14:26:55 -05:00
Luke Lovett
ab14c6f728 BUMP 2.9.2 2016-02-01 14:20:04 -08:00
Luke Lovett
0f01056114 Changelog for PyMongo 2.9.2 2016-02-01 14:15:57 -08:00
Bernie Hackett
c085738fed PYTHON-1048 - Auth module fixes for python 3.1 2016-01-29 14:14:28 -08:00
Bernie Hackett
8b7a13629b PYTHON-1044 - Fix up unknown BSON type handing 2016-01-29 13:40:03 -08:00
Bernie Hackett
0a5ef8de6e PYTHON-1040 - Add and use client._disconnect 2016-01-15 10:53:24 -08:00
Daniel Playfair Cal
65c0aed610 Use codec options in aggregate command 2015-12-15 15:42:28 -08:00
Bernie Hackett
8ee51cf438 PYTHON-1021 - Support git describe based server versions 2015-12-15 15:40:35 -08:00
Bernie Hackett
2d4a1d8d4c Start work on 2.9.2 2015-12-15 11:03:53 -08:00
Bernie Hackett
8aec72b74b BUMP 2.9.1 2015-11-17 15:52:33 -08:00
Bernie Hackett
caa3cff558 Minor changelog updates 2015-11-17 15:41:51 -08:00
Luke Lovett
616f44f3c2 PYTHON-1013 - Only reraise PyExc_Exceptions as InvalidBSON in get_value. 2015-11-17 11:23:56 -08:00
Bernie Hackett
0519c5d763 Changelog updates for PyMongo 2.9.1 2015-11-16 16:55:08 -08:00
Bernie Hackett
f88c5ff2bc Fix a bulk operations test for MongoDB 3.2 behavior change 2015-11-15 22:23:52 -08:00
aherlihy
03f01774fa PYTHON-1010 write_dict now checks the error indicator when exiting iteration loop 2015-11-13 11:07:08 -05:00
Bernie Hackett
2c46afed9b Start work on 2.9.1 2015-11-13 10:27:49 -05:00
Bernie Hackett
5936007de6 BUMP 2.9 2015-09-30 16:31:54 -07:00
Bernie Hackett
13d80ced6b Claim support for python 3.5 2015-09-30 16:15:45 -07:00
Bernie Hackett
a57c63355e Silence pointless compiler warnings 2015-09-29 19:00:17 -07:00
Bernie Hackett
fb01841f50 PYTHON-996 - Adjust regex tests for python 3.5 2015-09-22 14:46:33 -07:00
Bernie Hackett
6440ebe89a Fix a test under python 3.5 2015-09-22 13:14:05 -07:00
Bernie Hackett
e0bfbaa0be Fix up profiling tests for MongoDB 3.1.x 2015-09-11 16:59:39 -07:00
Luke Lovett
819febc041 PYTHON-977 - avoid 'b' string prefix for Python version compatibility in tests. 2015-09-04 16:01:44 -07:00
Luke Lovett
18054a19fc PYTHON-977 - Fix __hash__ method on BSON types that inherit from Python builtin types.
In Python 2, objects automatically inherit the __hash__ of their parent
class. In Python 3, objects that override __eq__ do not automatically inherit
__hash__, so these objects were not hashable under Python 3.  Additionally,
mutable BSON types and types that overide __eq__ but did not explicitly define
__hash__ had broken __hash__ methods under Python 2.  This commit unifies the
hashing behavior between Python versions and fixes the __hash__ methods such
that two BSON objects hash the same only if they are equal.

N.B.: bson.code.Code and bson.regex.Regex are no longer hashable under Python 2
because they are mutable.
2015-09-04 14:56:43 -07:00
Bernie Hackett
b31399b3b0 PYTHON-980 - Document deprecation of the eval command 2015-08-21 17:24:55 -07:00
Bernie Hackett
106b484578 PYTHON-974 - Use appropriate hash comparators for sensitive functions 2015-08-14 13:38:47 -07:00
Bernie Hackett
2feba07edf Post rc0 changes 2015-08-14 13:38:12 -07:00
Bernie Hackett
17e04afebc BUMP 2.9rc0 2015-08-05 20:37:19 -07:00
Bernie Hackett
283bfa364f Fix up copyright dates 2015-08-05 19:30:30 -07:00
Bernie Hackett
c8c4822ba5 PYTHON-967 - Call print as a function in README 2015-08-05 18:23:19 -07:00
Bernie Hackett
dd042a3957 Link to issues resolved 2015-08-05 18:21:12 -07:00
Bernie Hackett
29652d2460 Note that not all deprecated features raise DeprecationWarning 2015-07-17 13:17:23 -07:00
Bernie Hackett
699b6d3cdf PYTHON-966 - 3.0 compatibility for read preference options 2015-07-17 12:41:18 -07:00
Bernie Hackett
23cb737cae PYTHON-832 - Re-enable copy_database auth tests for MongoDB 3.0+. 2015-07-15 15:28:12 -07:00
Bernie Hackett
3bb6f0865d Changelog for 2.9. 2015-07-15 14:52:46 -07:00
Bernie Hackett
d19786423f Fix high availability examples to align with 3.0 2015-07-15 09:49:03 -07:00
Bernie Hackett
92968aad19 PYTHON-965 Backport MongoClient.address.
This change also deprecates MongoClient.host and MongoClient.port,
which are removed in PyMongo 3.
2015-07-14 13:13:12 -07:00
Bernie Hackett
c11aae4071 Fix $comment tests for MongoDB 3.1.x. 2015-07-13 17:19:53 -07:00
Bernie Hackett
89b7a9d5af PYTHON-963 - A few more doctest fixes. 2015-07-13 16:24:53 -07:00
Bernie Hackett
d312f8787d PYTHON-887 - Deprecate all features removed in 3.0 2015-07-13 14:56:32 -07:00
Luke Lovett
a8ad0656b0 PYTHON-963 - Fix doctests for documentation when running against MongoDB 2.6.x, 3.0.x. 2015-07-10 17:07:49 -07:00
Guillaume Gelin
e6ce54680b Use a standard version variable name (specified by PEP8)
See https://www.python.org/dev/peps/pep-0008/#version-bookkeeping
2015-06-30 15:08:15 -07:00
Bernie Hackett
ff79cbacd6 PYTHON-954 - Update links and text for TxMongo 2015-06-30 15:04:31 -07:00
Bernie Hackett
e4c8d17a8f PYTHON-955 - Backport connect option. 2015-06-25 11:47:42 -07:00
Bernie Hackett
8431379afa PYTHON-884 - Add 3.0 migration guide. 2015-06-24 17:03:06 -07:00
aherlihy
8f6c0ec68c PYTHON-950 - Backport CRUD write methods. 2015-06-23 11:21:27 -07:00
aherlihy
219e8a084e PYTHON-949 Support CodecOptions in message.py. 2015-06-17 12:27:48 -07:00
Bernie Hackett
c10a098c2c PYTHON-951 - Use bytes in GridFS tests. 2015-06-16 16:54:02 -07:00
Bernie Hackett
1c6ed1b351 PYTHON-951 - Raise CorruptGridFile for truncated chunks. 2015-06-16 16:22:45 -07:00
Bernie Hackett
331347871c PYTHON-888 - Skip UUID test on python 2.4. 2015-06-16 13:40:41 -07:00
aherlihy
4d6e8637bf PYTHON-888 Add codec_options support to the bson module. 2015-06-16 12:51:36 -07:00
Bernie Hackett
3487b4bfa1 PYTHON-834 - Add option to disable match_hostname. 2015-06-16 12:47:23 -07:00
Bernie Hackett
9b1ac9717f PYTHON-882 - Backport get_database, get_collection, and with_options.
This change comes with a small backward breaking change in the
behavior of client.document_class in exchange for enhanced
configuration flexibility. See changelog.rst for details and
examples.
2015-06-11 14:54:04 -07:00
aherlihy
4cf64b6170 PYTHON-945 Remove validation of the OP_REPLY "startingFrom" field 2015-06-08 16:20:24 -07:00
aherlihy
c6abb18b2e PYTHON-940 - Unhelpful and pretty wrong error message. 2015-06-08 15:55:26 -07:00
Bernie Hackett
fc9a053c90 PYTHON-889 - Document change in cursor support. 2015-06-08 15:17:29 -07:00
aherlihy
d19f4996ca PYTHON-889 The aggregate method should always return CommandCursor when passed the cursor option. 2015-06-08 15:05:38 -07:00
aherlihy
5ffe8d44bc PYTHON-933 - "maxPoolSize=0" allowed, causes hang 2015-06-08 14:49:48 -07:00
aherlihy
e405ef91cf PYTHON-885 - Support maxPoolSize URI option. 2015-06-08 14:11:00 -07:00
Bernie Hackett
a08f16d9dc PYTHON-890 - Minor doc fixes. 2015-06-08 13:38:12 -07:00
aherlihy
6e4608b9cd PYTHON-890 - Support localThresholdMS URI option. 2015-06-08 13:26:48 -07:00
behackett
313e21ab4f PYTHON-886 - Add missing doc target. 2015-06-03 17:02:01 -04:00
aherlihy
f6c28646d6 PYTHON-886 - Add CRUD API find / find_one options. 2015-06-03 16:49:03 -04:00
Bernie Hackett
d70578f650 PYTHON-881 - Backport CodecOptions class from 3.x. 2015-05-26 19:23:40 -07:00
Bernie Hackett
f2c01af265 Fix indentation in docs. 2015-05-26 12:34:55 -07:00
aherlihy
666a31438b PYTHON-883 - Backport CursorType from 3.x 2015-05-26 12:21:41 -07:00
Bernie Hackett
32c06e0494 PYTHON-879 - Backport new read preference classes from 3.x. 2015-05-22 10:36:06 -07:00
Bernie Hackett
8d072d59e8 PYTHON-880 - Backport WriteConcern class from 3.x. 2015-05-21 14:38:44 -07:00
Bernie Hackett
43d26ad73d Add links in client and connection docs. 2015-05-20 16:06:17 -07:00
aherlihy
c7f1546358 PYTHON-911 - Add database.client alias 2015-05-20 15:52:58 -07:00
Bernie Hackett
0f9ac50274 Start work on 2.9. 2015-05-13 13:43:01 -07:00
Bernie Hackett
c284c51c87 BUMP 2.8.1 2015-05-11 13:23:38 -07:00
Bernie Hackett
11a3d54f60 Changelog for PyMongo 2.8.1. 2015-05-11 10:57:42 -07:00
Bernie Hackett
5ed6a24086 PEP8 2015-05-07 16:11:46 -07:00
nahumoz@gmail.com
1152f89430 Add log4mongo-python
Log4mongo-python is actively maintained and upaded to the latest pymongo versions (2.8 and 3.X).
2015-05-07 16:03:41 -07:00
Bernie Hackett
93bef6eb23 PYTHON-915 - More robust latency test. 2015-05-07 15:53:34 -07:00
Bernie Hackett
d8f0e4c000 PYTHON-894 - More doc clarifications for alive. 2015-05-07 15:00:37 -07:00
Bernie Hackett
90efec37ff PYTHON-915 - Allow 0 for secondaryAcceptableLatencyMS. 2015-05-06 18:13:12 -07:00
Bernie Hackett
9bf46d2bb1 PYTHON-920, PYTHON-921 - Fix metadata helpers with direct connection.
With this change metadata helpers (database_names, collection_names,
options, and index_information) no longer require a non-primary read
preference or the slave_okay option to run them against a directly
connected secondary or slave. This was the easiest way to make the
driver behave consistently across MongoDB versions and helpers. This
is also consistent with the server selection spec, though we are not
implementing that spec for PyMongo 2.x.
2015-05-06 09:14:55 -07:00
A. Jesse Jiryu Davis
4897c51090 PYTHON-903 - Discover primary despite auth errs on recovering members. 2015-05-04 13:58:55 -04:00
behackett
a73d3cbdab PYTHON-841 FAQ entry for key order and subdocument matching. 2015-04-28 17:29:06 -04:00
behackett
5d8194d0f3 PYTHON-913 - Fix test for old python versions. 2015-04-28 17:26:13 -04:00
behackett
b172a1f1a9 PYTHON-913 - Suppress read preference warning under MongoClient. 2015-04-28 15:45:17 -04:00
A. Jesse Jiryu Davis
8e9bd739b0 Longer wait queue timeout in test_auth_network_error. 2015-04-24 22:25:51 -04:00
A. Jesse Jiryu Davis
8c5e547274 PYTHON-903 - Don't *require* a network error in auth test. 2015-04-23 22:08:23 -04:00
Bernie Hackett
4c39f1a99f PYTHON-903 - Can't use "as" in PyMongo 2.x. 2015-04-23 13:55:24 -07:00
A. Jesse Jiryu Davis
5c98b1ebf3 PYTHON-903 - Survive auth failures on secondaries. 2015-04-23 16:42:03 -04:00
A. Jesse Jiryu Davis
9fc992b423 PYTHON-903 - Inc semaphore after network err with auth.
There was a semaphore leak in MongoReplicaSet.__socket, though not critical.
After getting disconnected during auth, it discards the server's whole pool.
Then it wakes the monitor and creates a new pool, recovering from the leak.
That's fine unless, at the moment we wake the monitor, we're unlucky and it's
just started running using the previous pool (due to periodic monitoring or
whatever). In that case it can't get a socket, and tells the main thread it
thinks the primary's down *again*. Thus we threw two errors instead of one, but
we still recover eventually.
2015-04-23 16:42:02 -04:00
A. Jesse Jiryu Davis
f05e800820 Fix test_auth_network_error for replsets.
We added a user then tried to admin with a disconnected client. During
reconnect it may try to log in with a secondary, which hasn't always
replicated the credentials yet. This change uses the global test creds
instead of creating new ones, so replication doesn't confound the test.
2015-04-23 15:24:23 -04:00
behackett
18711cae93 Add required whitespace. 2015-04-19 10:43:15 -07:00
Vladimir Savin
12cefd69c0 Small fix documentation aggregation
Updated the example of Grouping
2015-04-19 10:41:36 -07:00
A. Jesse Jiryu Davis
22cf7f2918 PYTHON-894 - Can't use addCleanup in PyMongo 2 tests. 2015-04-17 18:54:52 -04:00
A. Jesse Jiryu Davis
2ef99ef692 PYTHON-894 - Can't use addCleanup in PyMongo 2 tests. 2015-04-17 17:36:01 -04:00
A. Jesse Jiryu Davis
f18b3644c2 PYTHON-894 - Set cursor.alive False after final batch.
Even with this change, "next" can raise StopIteration even though
"alive" is True. For example if batch size is 2 and there are 4
documents in the result set, then after the 4th document "alive" is True
but "next" raises StopIteration.
2015-04-17 16:19:03 -04:00
Bernie Hackett
773a8dff0b PYTHON-842 - Fix tests for python with no ssl module. 2015-04-14 11:17:32 -07:00
Bernie Hackett
eaaa54b903 PYTHON-893 - Fix application of SON manipulators in CommandCursor. 2015-04-14 11:07:51 -07:00
Bernie Hackett
eda1e771f6 PYTHON-842 - SSL URI config support.
This commit cleans up and builds on the work in commit
cc943f176c.
2015-04-14 10:59:00 -07:00
Len Buckens
cc943f176c allow ssl_cert_reqs options to be passed as string 2015-04-13 12:27:05 -07:00
Bernie Hackett
2598869d26 PYTHON-864 - Support RFC-3339 offset format for $date. 2015-04-13 12:09:58 -07:00
Bernie Hackett
43347f61f1 Start work on 2.8.1. 2015-04-13 11:22:45 -07:00
Bernie Hackett
7f4c0588bc BUMP 2.8 2015-01-28 13:19:25 -08:00
Bernie Hackett
0687e9b656 PYTHON-832 - Disable copy_database auth tests with MongoDB 2.7.2+ 2015-01-28 11:48:42 -08:00
Bernie Hackett
84c34a3d45 Remove redundant test_copy_db_scram_sha_1. 2015-01-28 11:48:27 -08:00
Bernie Hackett
720a141227 Update MongoDB version references from 2.8 to 3.0. 2015-01-24 09:21:32 -08:00
Bernie Hackett
4ec3f880e1 PYTHON-830 - Fix bad uses of _get_wc_override. 2015-01-24 09:00:25 -08:00
Bernie Hackett
4acb891473 Revert "PYTHON-830 - Fix bad uses of _get_wc_override."
This reverts commit 19753f3897.
2015-01-23 18:18:07 -08:00
Bernie Hackett
19753f3897 PYTHON-830 - Fix bad uses of _get_wc_override. 2015-01-23 14:43:40 -08:00
Bernie Hackett
82af07b9c8 Remove "text" command read preference test. 2015-01-20 10:08:15 -08:00
Bernie Hackett
6b218b5120 Version -> + 2015-01-20 10:08:01 -08:00
Bernie Hackett
0b715cff2e BUMP 2.8rc2 2014-12-24 11:38:35 -08:00
Bernie Hackett
83f53499ae Fix auth DeprecationWarning test. 2014-12-23 15:37:47 -08:00
A. Jesse Jiryu Davis
9e6a267854 PYTHON-786: Silence DeprecationWarnings in test_master_slave_connection. 2014-12-21 17:11:07 -06:00
A. Jesse Jiryu Davis
61f526a2c5 PYTHON-783 Silence copy_database DeprecationWarnings in tests. 2014-12-21 17:11:07 -06:00
A. Jesse Jiryu Davis
31b83bc0e0 PYTHON-807 Silence DeprecationWarnings in getlasterror tests. 2014-12-21 17:11:07 -06:00
A. Jesse Jiryu Davis
8e794bc8fe PYTHON-807 Avoid DeprecationWarning in GridIn.close(). 2014-12-21 17:08:03 -06:00
A. Jesse Jiryu Davis
753356a723 PYTHON-807 Deprecate Database.error() and related methods. 2014-12-19 19:56:02 -05:00
James Root
2aae624dda Updated minimongo repository location 2014-12-18 16:19:37 -08:00
Bernie Hackett
a73e6cfb13 PYTHON-806 - Always use command cursor 'ns' value for OP_GET_MORE 2014-12-18 13:39:38 -08:00
Bernie Hackett
f3b6abf622 PYTHON-796 - Support listCollections and listIndexes command cursors 2014-12-18 12:12:57 -08:00
A. Jesse Jiryu Davis
be1a7be639 PYTHON-799 Python 2.4 compatibility in test_kill_cursors_warning. 2014-12-17 15:17:59 -05:00
A. Jesse Jiryu Davis
69beec6ca6 PYTHON-799 RS client's close_cursor shouldn't lock client.
We still take a lock in Pool.get_socket and maybe_return_socket, but in my tests
it doesn't seem possible to deadlock the GC from Pool.
2014-12-17 14:12:36 -05:00
A. Jesse Jiryu Davis
d195c7c70d PYTHON-799 Warn when MongoClient can't close a cursor. 2014-12-17 14:12:36 -05:00
A. Jesse Jiryu Davis
8ebd553d5a PYTHON-799 Avoid deadlock in Cursor destructor with PyPy. 2014-12-17 14:11:54 -05:00
A. Jesse Jiryu Davis
e93c2ac72c Wrong docstring for MongoReplicaSetClient.close_cursor. 2014-12-13 10:09:44 -05:00
A. Jesse Jiryu Davis
967a243469 Unused argument in MasterSlaveConnection._send_message. 2014-12-13 10:04:07 -05:00
A. Jesse Jiryu Davis
4a085f1d33 Fix formatting in changelog. 2014-12-12 15:24:35 -05:00
Bernie Hackett
5c26dab41a Version -> + 2014-12-02 16:19:01 -08:00
Bernie Hackett
698e099969 BUMP 2.8rc1 2014-12-02 16:07:01 -08:00
Bernie Hackett
1398a4b782 Raise if nonce or server signature don't match. 2014-12-02 12:08:18 -08:00
Bernie Hackett
86e85ce715 PYTHON-795 - Fix password handling for None and the empty string. 2014-12-02 08:45:37 -08:00
Bernie Hackett
807c6797e1 PYTHON-792 - Update create collection and index docs. 2014-11-24 15:42:43 -08:00
Bernie Hackett
e9e764c4f3 Add Heewa Barfchin to contributors. 2014-11-24 11:05:33 -08:00
Bernie Hackett
cf791ca74e Update changelog to mention new BSON helpers. 2014-11-24 11:04:12 -08:00
Bernie Hackett
e77607f1a4 Document uuid_subtype parameter for BSON.encode/decode. 2014-11-21 16:47:30 -08:00
Bernie Hackett
cfbaf7ef95 Document to uuid_subtype parameter for decoders. 2014-11-21 16:44:41 -08:00
Bernie Hackett
0be172bdf8 Fix up docs for decode_(file_)iter. 2014-11-21 16:33:44 -08:00
Bernie Hackett
fd2d8face2 Allow decode_(file_)iter to use C extensions. 2014-11-21 16:26:19 -08:00
Bernie Hackett
7858dcb868 Add compile_re support to decode_(file_)iter. 2014-11-21 15:30:50 -08:00
Heewa Barfchin
5c4556b013 Add generator versions of decode_all in bson.
When decoding large collections of bson documents, the python representation
of dicts are time and space costly, so it's sometimes useful to generate and
consume the documents iteratively. This patch adds two new functions to do
that: decode_iter and decode_file_iter. The first is given all the bson data,
but yields one document at a time, while the second reads from a file object
enough to yield one document at a time (to avoid reading in an entire file).
2014-11-21 15:22:24 -08:00
Bernie Hackett
80bcdb3156 PYTHON-791 - Fix JSON support for Timestamp. 2014-11-21 09:26:37 -08:00
A. Jesse Jiryu Davis
efb2c2a9dc PYTHON-789 Clarify valid ObjectId input. 2014-11-20 16:51:52 -05:00
Mieszko
1cf2f166e9 Two typos 2014-11-19 17:03:13 -05:00
A. Jesse Jiryu Davis
1cf4e15442 Version -> + 2014-11-19 17:03:13 -05:00
Bernie Hackett
21ef41346a BUMP 2.8rc0 2014-11-12 09:55:38 -08:00
Bernie Hackett
1c2f0cdb30 Changelog and related fixes. 2014-11-12 09:12:17 -08:00
Bernie Hackett
539d6f3d07 Fix a typo in the changelog. 2014-11-11 15:08:35 -08:00
Bernie Hackett
5755079dc4 PYTHON-778 - Document URI quoting rules. 2014-11-11 14:43:09 -08:00
Bernie Hackett
71bf8cf6e7 Changelog for PyMongo 2.8. 2014-11-11 14:24:49 -08:00
A. Jesse Jiryu Davis
8e119d2b8c Typo in README.rst. 2014-11-10 21:36:32 -05:00
A. Jesse Jiryu Davis
eb5d4f61bf Note that mongos does not support copydb with auth. 2014-11-10 19:48:42 -05:00
A. Jesse Jiryu Davis
cd3d37b43a Skip copydb auth test with mongos. 2014-11-10 19:42:41 -05:00
A. Jesse Jiryu Davis
0f1f99b52b PYTHON-784 Deprecate start_request. 2014-11-10 18:43:41 -05:00
A. Jesse Jiryu Davis
4579303838 More thorough test_last_error_options. 2014-11-10 18:33:22 -05:00
A. Jesse Jiryu Davis
e1826051f4 Fix TestBulkWriteConcern for MongoDB 2.8. 2014-11-10 18:33:22 -05:00
A. Jesse Jiryu Davis
fd22f89f9e Fix test_last_error_options for MongoDB 2.8. 2014-11-10 18:33:22 -05:00
A. Jesse Jiryu Davis
5e60982cf8 nit 2014-11-10 18:18:22 -05:00
A. Jesse Jiryu Davis
8925aec75d PYTHON-720 Explain deprecations and compatibility policy. 2014-11-10 18:18:22 -05:00
A. Jesse Jiryu Davis
56633d8bb5 PYTHON-786 Deprecate MasterSlaveConnection. 2014-11-10 17:46:36 -05:00
A. Jesse Jiryu Davis
facbb99611 PYTHON-783 Deprecate copy_database. 2014-11-10 11:17:20 -05:00
A. Jesse Jiryu Davis
52f2314947 More reliable TestClientAuth.test_copy_db. 2014-11-07 23:09:42 -05:00
A. Jesse Jiryu Davis
0b7b51975e PYTHON-777 Make copy_database work with SCRAM-SHA-1.
See doc/examples/copydb.rst for details.
2014-11-07 17:13:40 -05:00
A. Jesse Jiryu Davis
f787165d43 PYTHON-782 Fix readchunk() for disconnected GridOut. 2014-11-06 22:18:04 -05:00
A. Jesse Jiryu Davis
bd895ca079 PYTHON-782 Demonstrate error from readchunk() of disconnected GridOut. 2014-11-06 22:18:04 -05:00
Bernie Hackett
49ff70c39e Add Sergey Azovskov to contributors. 2014-11-02 08:10:28 -08:00
Bernie Hackett
91e25bf7ab Add version information to docstring. 2014-11-02 08:07:55 -08:00
Bernie Hackett
6a39f811b2 Manipulate defaults to False in find_and_modify. 2014-11-02 08:06:33 -08:00
Sergey Azovskov
5ccd02653a Added support for manipulate param in find_and_modify for consistency with find method 2014-11-02 08:02:11 -08:00
Bernie Hackett
c7bbafe373 Add cannium to contributors. 2014-11-02 07:58:46 -08:00
Bernie Hackett
3f0e4f6093 Add version information to docstrings. 2014-11-02 07:55:11 -08:00
Bernie Hackett
85a42eaf91 Use connect=False when authenticating on slaves. 2014-11-02 07:53:00 -08:00
Can ZHANG
d3c81be2ac Add close, server_info, _cache_credentials, _purge_credentials to master_slave_connection 2014-11-02 07:47:04 -08:00
Bernie Hackett
d8d15e4710 Update travis.yml 2014-10-31 16:08:00 -07:00
Bernie Hackett
b5f94974c4 Fix MasterSlaveConnection insert test. 2014-10-29 14:11:35 -07:00
Bernie Hackett
d3e88ee8ed PYTHON-768 - Support authMechanismProperties.
This change also deprecates the gssapiServiceName option, which
is replaced by authMechanismProperties=SERVICE_NAME:<service name>.
2014-10-29 13:37:05 -07:00
A. Jesse Jiryu Davis
c101ed036c Test insert with MasterSlaveConnection. 2014-10-29 11:53:00 -04:00
A. Jesse Jiryu Davis
7c5d8b6fc1 Delete two unreliable master-slave tests.
These tests were guaranteed to fail occasionally, but they didn't test
a driver feature anyway.
2014-10-29 11:53:00 -04:00
Bernie Hackett
37c5c6b99a PYTHON-781 - Fix tests for multiple storage engines.
Remove a few seemingly pointless test cases that
aren't portable across server versions, storage engines,
etc.
2014-10-28 15:30:25 -07:00
A. Jesse Jiryu Davis
c70b79445e GridFS.find_one docstring format. 2014-10-28 16:22:43 -04:00
A. Jesse Jiryu Davis
10ba3b46a7 Test GridFS.find_one. 2014-10-28 15:53:36 -04:00
A. Jesse Jiryu Davis
7c25f933f2 Style. 2014-10-28 15:53:36 -04:00
ximing
69b5155814 Add find_one() method for gridfs. 2014-10-28 09:13:55 -04:00
Bernie Hackett
f6597e46a8 PYTHON-644 - Restore dict.copy() to avoid Jython issues. 2014-10-27 18:04:53 -07:00
A. Jesse Jiryu Davis
a657a263b5 PYTHON-703 Fix tests for Python 2.4. 2014-10-24 13:09:46 -04:00
A. Jesse Jiryu Davis
0cdb43fcc5 PYTHON-703 Remove slow SON.__contains__ method.
__setitem__ must now check self.__keys to avoid setting keys twice during
"copy.copy(son_obj)".
2014-10-24 13:07:55 -04:00
Don Mitchell
26f3f40fc9 Allow destructive ops during son iteration
and let python handle (identical to iter on list)
2014-10-24 13:07:55 -04:00
Don Mitchell
01b499850c Don't unnecessarily copy the key list
Conflicts:
	doc/contributors.rst
	test/test_son.py
2014-10-24 13:07:55 -04:00
A. Jesse Jiryu Davis
c18501c596 PYTHON-706 Use fastest SCRAM-SHA-1 implementation available. 2014-10-24 13:03:44 -04:00
A. Jesse Jiryu Davis
ab1c2bc894 PYTHON-706 Mention that backports.pbkdf2 provides the best performance.
(cherry picked from commit a7db21a)
2014-10-23 21:27:17 -04:00
behackett
1307969f6e PYTHON-706 - Optimize XOR in SCRAM HI. 2014-10-23 21:05:56 -04:00
behackett
45d058f123 PYTHON-644 - Send w=1 to the server if explicitly provided. 2014-10-23 17:08:45 -04:00
A. Jesse Jiryu Davis
a82d2b62ce Bugfix an AttributeError in my PYTHON-764 implementation. 2014-10-23 16:46:14 -04:00
A. Jesse Jiryu Davis
3ca47b804b PYTHON-764 Update auth examples for MongoDB 2.8. 2014-10-23 08:28:09 -04:00
A. Jesse Jiryu Davis
e3d6510761 PYTHON-764 SCRAM-SHA-1 automatic upgrade / downgrade. 2014-10-23 08:28:09 -04:00
A. Jesse Jiryu Davis
bf091b8d22 Increase WTimeout in test_gridfs_replica_set. 2014-10-20 09:06:55 -04:00
A. Jesse Jiryu Davis
0e5780751a PYTHON-736 Delete racy test_exhaust_getmore_server_error. 2014-10-17 17:11:09 -04:00
Adam Comerford
e9240a8bee Update grid_file.py
Comments need to reflect 256 to 255 kb default chunk size change to propagate to docs
2014-10-17 13:45:45 -04:00
A. Jesse Jiryu Davis
eb25125d64 PYTHON-757 Warn against installing third-party "bson" package.
Do **not** install the "bson" package. PyMongo comes with its own bson package;
doing "pip install bson" installs a third-party package that is incompatible
with PyMongo.
2014-10-09 16:56:10 -04:00
A. Jesse Jiryu Davis
2ef85956b8 PYTHON-747 Fix TypeError in Pool.__del__ during shutdown. 2014-10-09 16:15:29 -04:00
A. Jesse Jiryu Davis
2c6ecb490a PYTHON-766 Python 2.4-compatible test for parsing mongos errors. 2014-10-09 11:30:47 -04:00
A. Jesse Jiryu Davis
0397ab71bb PYTHON-755 Don't wait forever for monitor threads during interpreter shutdown. 2014-10-08 15:07:34 -04:00
A. Jesse Jiryu Davis
41dc866e33 PYTHON-749 Handle floating-point chunkSize in GridOut. 2014-10-08 12:27:21 -04:00
A. Jesse Jiryu Davis
fedad1162e PYTHON-749 Test that GridOut handles chunkSize as a float. 2014-10-08 12:25:46 -04:00
A. Jesse Jiryu Davis
2e74187e19 PYTHON-766 Fix KeyError when parsing certain mongos error responses. 2014-10-07 19:38:34 -04:00
A. Jesse Jiryu Davis
90098d3844 PYTHON-766 Demonstrate a bug parsing an error message from mongos. 2014-10-07 19:38:34 -04:00
Bernie Hackett
7dc7145800 Debug exhaust cursor test. 2014-10-06 14:04:20 -07:00
behackett
225cd39187 PYTHON-761 - Use listCollections to helper methods 2014-10-02 16:21:11 -07:00
behackett
b74aafb577 PYTHON-762 - Use listIndexes for index_information 2014-10-02 16:20:21 -07:00
behackett
29d5bca15e PYTHON-763 - Bump MAX_SUPPORTED_WIRE_VERSION to 3. 2014-10-02 16:16:58 -07:00
A. Jesse Jiryu Davis
5538c87552 Dead test code. 2014-10-02 09:32:24 -04:00
Bernie Hackett
f8e6d36c8a PYTHON-493 - Add **kwargs to Database.dereference 2014-09-26 16:18:47 -07:00
Bernie Hackett
be12ae5ad8 Can't test ssl.CERT_NONE without an SSL module.
Python 2.4, 2.5, Jython, etc.
2014-09-26 15:34:07 -07:00
Bernie Hackett
d6593fc24b PYTHON-679 - Add socketKeepAlive option. 2014-09-26 15:12:05 -07:00
Bernie Hackett
5d3e294b93 PYTHON-693 - Fix parsing of default values for keyword args. 2014-09-26 08:10:28 -07:00
Bernie Hackett
5d2ea1f994 PYTHON-753 - Add "How To Ask For Help" in README.rst 2014-09-25 13:18:30 -07:00
Bernie Hackett
70aaf0dc52 PYTHON-739 - Add namespace to command failure message. 2014-09-24 13:55:08 -07:00
Bernie Hackett
1e4b0c5a12 PYTHON-752 - Fix escaping in __simple_command. 2014-09-24 12:28:56 -07:00
Bernie Hackett
0fb2fcfac0 PYTHON-700 - Support subclassing of son manipulators 2014-09-24 09:50:31 -07:00
A. Jesse Jiryu Davis
9dda1346dd Test connecting to standalone or RS from the same mod_wsgi script. 2014-09-24 09:52:41 -04:00
Luke Lovett
083b1530ae PYTHON-765 Allow tests to use an existing user when running with auth enabled. 2014-09-19 22:38:53 +00:00
A. Jesse Jiryu Davis
ad3a03ab56 Remove unreliable TestPoolSocketSharingThreads test.
It relies on a find_one being faster on the server than a particular JS function,
which is not reliable on virtual hardware. Furthermore the code it's testing is
very well exercised for both threads and greenlets in _TestMaxPoolSize.
2014-09-19 15:40:22 -04:00
Bernie Hackett
f66441514d Skip copy_database fromhost test on buggy mongos versions. 2014-09-18 16:27:15 -07:00
Bernie Hackett
69a08095d6 PYTHON-759 - Fix ISO-8601 support for python 2.4 and 2.5 2014-09-18 10:10:50 -07:00
Bernie Hackett
9e7fd4865a PYTHON-759 - Support $date as ISO-8601 or $numberLong 2014-09-17 13:36:43 -07:00
Bernie Hackett
29c885311f PYTHON-708 Support $undefined and $numberLong extended JSON types. 2014-09-17 13:33:19 -07:00
Bernie Hackett
1ac607c447 PYTHON-314, PYTHON-744 - Hint by index name, count with hint.
This change introduces two closely related features. Cursor.hint
now accepts the name of an index as an alternative to passing
the index spec. Cursor.count will now pass the hint, if one was
specified, to the count command. Count with hint is only supported
by MongoDB 2.6 and newer.
2014-09-12 13:50:04 -07:00
Bernie Hackett
91622e62d5 Fix index tests for MongoDB 2.7.x explain output 2014-09-05 15:00:10 -07:00
Bernie Hackett
93e7db4ec3 PYTHON-706 - SCRAM-SHA-1 support for PyMongo 2.x 2014-09-05 06:48:59 -07:00
Bernie Hackett
793438e681 PYTHON-754 - Deprecate dropDups / drop_dups
Also skip the related test when connected to MongoDB >= 2.7.
2014-08-29 16:48:00 -07:00
Luke Lovett
7decdd8a40 PYTHON-751 Move all auth tests to the test_auth module. 2014-08-29 22:51:39 +00:00
Bernie Hackett
33d0c702a5 Version -> + 2014-08-14 09:46:29 -07:00
Bernie Hackett
0d4a2ef28a BUMP 2.7.2 2014-07-29 14:36:57 -07:00
Bernie Hackett
9ca8ad7fc9 Changelog for version 2.7.2. 2014-07-25 10:56:39 -07:00
Bernie Hackett
d32016274b PYTHON-736 - Don't close sockets on OperationFailure
This also speeds up returning exhaust sockets to the pool
when the server returns an error and fixes the tests to
run against all MongoDB versions we test against.
2014-07-24 13:22:23 -07:00
A. Jesse Jiryu Davis
9ad421a58a PYTHON-736 Fix exhaust cursor error-handling.
Connection-pool semaphore leak on server error when
creating or iterating an exhaust cursor.
2014-07-23 16:35:01 -07:00
Bernie Hackett
46a7df09bd PYTHON-738 - Clarify versionchanged line for bulk insert. 2014-07-22 16:17:14 -07:00
A. Jesse Jiryu Davis
b293b7735b import style 2014-07-15 11:08:43 -04:00
A. Jesse Jiryu Davis
e1a7bc5058 PYTHON-732 Handle network errors when adding existing credentials to sockets. 2014-07-15 09:34:23 -04:00
Luke Lovett
a1f7a5487f PYTHON-714 Work around localhost exception issues in add_user when connected to MongoDB >= 2.7.1. 2014-07-02 17:11:20 +00:00
Bernie Hackett
952953d3a1 Fix tests under pypy3. 2014-06-21 22:44:33 -07:00
A. Jesse Jiryu Davis
e904f014d9 PYTHON-709 insert _id in document after applying non-copying SONManipulators. 2014-06-19 14:50:41 -04:00
A. Jesse Jiryu Davis
79df8d799a Revert "PYTHON-710, simplify SON's equality operator."
This reverts commit 551e1e3edf.
The change did not work as expected in Jython.
2014-06-19 14:10:24 -04:00
A. Jesse Jiryu Davis
6c68762960 Use modern 'distinct' syntax in tests.
The old syntax is now an error:
https://jira.mongodb.org/browse/SERVER-12642
2014-06-18 20:34:45 -04:00
A. Jesse Jiryu Davis
69f52d0cdf PYTHON-710, simplify SON's equality operator. 2014-06-18 17:50:55 -04:00
A. Jesse Jiryu Davis
686c8fae49 PYTHON-710, SON.to_dict shouldn't change original data. 2014-06-18 17:49:57 -04:00
A. Jesse Jiryu Davis
91a56702cf PYTHON-710 test that SON.to_dict doesn't change data. 2014-06-18 17:44:12 -04:00
A. Jesse Jiryu Davis
fb207af4cf PYTHON-712 ObjectId.is_valid(None) should be False. 2014-06-18 16:19:58 -04:00
behackett
2dc840955a PYTHON-705 - Fix Bulk API legacy upsert _id compatibility
Versions of MongoDB previous to 2.6 only return the upserted
field for an upsert operation if the _id value is an ObjectId.
This patch works around that issue to ensure nUpserted counts
are correct regardless of server version.
2014-06-06 15:37:31 -07:00
behackett
6fbb4c5307 Version -> + 2014-06-06 14:47:41 -07:00
Bernie Hackett
e959aad948 BUMP 2.7.1 2014-05-23 14:45:12 -07:00
Bernie Hackett
6baba92fcf Changelog for 2.7.1 2014-05-23 13:39:00 -07:00
Bernie Hackett
47825d9d39 PYTHON-697 - Fix upsert _id backward compatibility 2014-05-20 11:28:06 -07:00
Bernie Hackett
f739025e0c PYTHON-698 - Try encoding types with broken __getattr__ methods 2014-05-16 14:02:12 -07:00
Bernie Hackett
6991b73734 Fix a few tests for MongoDB 2.7.0 2014-05-12 14:07:25 -07:00
Bernie Hackett
04ff22e3c9 Various fixes for auth tests with old mongos versions. 2014-05-02 15:15:08 -07:00
Bernie Hackett
13cd9bee6f Fix a few tests with really old mongos versions. 2014-05-01 19:20:33 -07:00
Bernie Hackett
2cc560c671 PYTHON-696 - Fix remove_user for old mongos versions. 2014-05-01 15:41:16 -07:00
Bernie Hackett
b97b85f89a PYTHON-696 - Fix user and index creation with old mongos versions. 2014-05-01 14:27:29 -07:00
Jaroslav Semančík
15511b70d8 Added Jaroslav Semančík (girogiro) to contributors 2014-05-01 11:28:49 -07:00
Jaroslav Semančík
e299c044aa Fixed wrong Python object name for UTC 2014-05-01 11:28:48 -07:00
Bernie Hackett
32279986bd PYTHON-667 - Clarify drop_index behavior when an index does not exist. 2014-05-01 10:49:16 -07:00
Bernie Hackett
348bd628aa PYTHON-690 - Various fixes to indexing docstrings. 2014-05-01 09:59:09 -07:00
Bernie Hackett
9d47f1cd3d PYTHON-691 - Fix UserWarning command issues.
Don't raise UserWarning for helpers and internal calls to
commands that do not obey read preference.
2014-05-01 09:21:52 -07:00
Bernie Hackett
d703ebb832 PYTHON-684 - Use unordered bulk for unordered test. 2014-04-29 13:44:51 -07:00
Bernie Hackett
baed02fb11 PYTHON-685 - Fix rare resource leak in _cmessage 2014-04-29 13:14:55 -07:00
Bernie Hackett
f61b0e4f59 PYTHON-684 - Ignore wnote/jnote from legacy servers.
Stop unnecessarily raising OperationFailure in the Bulk API
when a pre-2.6 server returns a result with a wnote or jnote
field.
2014-04-29 11:41:35 -07:00
Bernie Hackett
7d55d77072 Version -> + 2014-04-29 11:40:26 -07:00
2124 changed files with 46137 additions and 597093 deletions

View File

@ -1,4 +0,0 @@
# do not notify until at least 100 builds have been uploaded from the CI pipeline
# you can also set after_n_builds on comments independently
comment:
after_n_builds: 100

View File

@ -1,16 +0,0 @@
#!/bin/bash
#
# Coverage combine merges (and removes) all the coverage files and
# generates a new .coverage file in the current directory.
set -eu
# Set up the virtual env.
. .evergreen/scripts/setup-dev-env.sh
uv sync --group coverage
source .venv/bin/activate
ls -la coverage/
coverage combine coverage/coverage.*
coverage html -d htmlcov

File diff suppressed because it is too large Load Diff

View File

@ -1,343 +0,0 @@
functions:
# Assume ec2 role
assume ec2 role:
- command: ec2.assume_role
params:
role_arn: ${aws_test_secrets_role}
duration_seconds: 3600
# Attach benchmark test results
attach benchmark test results:
- command: attach.results
params:
file_location: src/report.json
# Cleanup
cleanup:
- command: subprocess.exec
params:
binary: bash
args:
- .evergreen/scripts/cleanup.sh
working_dir: src
type: test
# Download and merge coverage
download and merge coverage:
- command: ec2.assume_role
params:
role_arn: ${assume_role_arn}
type: setup
- command: subprocess.exec
params:
binary: bash
args:
- .evergreen/scripts/download-and-merge-coverage.sh
- ${bucket_name}
- ${revision}
- ${version_id}
working_dir: src
silent: true
include_expansions_in_env:
- AWS_ACCESS_KEY_ID
- AWS_SECRET_ACCESS_KEY
- AWS_SESSION_TOKEN
type: test
- command: subprocess.exec
params:
binary: bash
args:
- .evergreen/combine-coverage.sh
working_dir: src
type: test
- command: subprocess.exec
params:
binary: bash
args:
- .evergreen/scripts/upload-coverage-report.sh
- ${bucket_name}
- ${revision}
- ${version_id}
working_dir: src
silent: true
include_expansions_in_env:
- AWS_ACCESS_KEY_ID
- AWS_SECRET_ACCESS_KEY
- AWS_SESSION_TOKEN
type: test
- command: s3.put
params:
remote_file: coverage/${revision}/${version_id}/htmlcov/index.html
aws_key: ${AWS_ACCESS_KEY_ID}
aws_secret: ${AWS_SECRET_ACCESS_KEY}
aws_session_token: ${AWS_SESSION_TOKEN}
bucket: ${bucket_name}
local_file: src/htmlcov/index.html
permissions: public-read
content_type: text/html
display_name: Coverage Report HTML
optional: "true"
type: setup
# Fetch source
fetch source:
- command: git.get_project
params:
directory: src
# Run server
run server:
- command: subprocess.exec
params:
binary: bash
args:
- .evergreen/just.sh
- run-server
- ${TEST_NAME}
working_dir: src
include_expansions_in_env:
- VERSION
- TOPOLOGY
- AUTH
- SSL
- ORCHESTRATION_FILE
- UV_PYTHON
- TOOLCHAIN_VERSION
- STORAGE_ENGINE
- REQUIRE_API_VERSION
- DRIVERS_TOOLS
- TEST_CRYPT_SHARED
- AUTH_AWS
- LOAD_BALANCER
- LOCAL_ATLAS
- NO_EXT
type: test
- command: expansions.update
params:
file: ${DRIVERS_TOOLS}/mo-expansion.yml
# Run tests
run tests:
- command: subprocess.exec
params:
binary: bash
args:
- .evergreen/just.sh
- setup-tests
- ${TEST_NAME}
- ${SUB_TEST_NAME}
working_dir: src
include_expansions_in_env:
- AUTH
- SSL
- AWS_ACCESS_KEY_ID
- AWS_SECRET_ACCESS_KEY
- AWS_SESSION_TOKEN
- COVERAGE
- UV_PYTHON
- LIBMONGOCRYPT_URL
- MONGODB_URI
- TOOLCHAIN_VERSION
- DISABLE_TEST_COMMANDS
- GREEN_FRAMEWORK
- NO_EXT
- COMPRESSORS
- MONGODB_API_VERSION
- REQUIRE_API_VERSION
- DEBUG_LOG
- DISABLE_FLAKY
- ORCHESTRATION_FILE
- OCSP_SERVER_TYPE
- VERSION
- IS_WIN32
- REQUIRE_FIPS
- TEST_MIN_DEPS
type: test
- command: subprocess.exec
params:
binary: bash
args:
- .evergreen/just.sh
- run-tests
working_dir: src
type: test
# Send dashboard data
send dashboard data:
- command: subprocess.exec
params:
binary: bash
args:
- .evergreen/scripts/perf-submission-setup.sh
working_dir: src
include_expansions_in_env:
- requester
- revision_order_id
- project_id
- version_id
- build_variant
- parsed_order_id
- task_name
- task_id
- execution
- is_mainline
type: test
- command: expansions.update
params:
file: src/expansion.yml
- command: subprocess.exec
params:
binary: bash
args:
- .evergreen/scripts/perf-submission.sh
working_dir: src
include_expansions_in_env:
- requester
- revision_order_id
- project_id
- version_id
- build_variant
- parsed_order_id
- task_name
- task_id
- execution
- is_mainline
type: test
# Setup system
setup system:
- command: subprocess.exec
params:
binary: bash
args:
- .evergreen/scripts/setup-system.sh
working_dir: src
include_expansions_in_env:
- is_patch
- project
- version_id
type: test
- command: expansions.update
params:
file: src/expansion.yml
# Teardown system
teardown system:
- command: subprocess.exec
params:
binary: bash
args:
- .evergreen/just.sh
- teardown-tests
working_dir: src
type: test
- command: subprocess.exec
params:
binary: bash
args:
- ${DRIVERS_TOOLS}/.evergreen/teardown.sh
working_dir: src
type: test
# Test numpy
test numpy:
- command: subprocess.exec
params:
binary: bash
args:
- .evergreen/just.sh
- test-numpy
working_dir: src
include_expansions_in_env:
- TOOLCHAIN_VERSION
- COVERAGE
type: test
# Upload coverage codecov
upload codecov:
- command: subprocess.exec
params:
binary: bash
args:
- .evergreen/scripts/upload-codecov.sh
working_dir: src
include_expansions_in_env:
- CODECOV_TOKEN
- build_variant
- task_name
- github_commit
- github_pr_number
- github_pr_head_branch
- github_author
- requester
- branch_name
type: test
# Upload coverage
upload coverage:
- command: ec2.assume_role
params:
role_arn: ${assume_role_arn}
type: setup
- command: s3.put
params:
remote_file: coverage/${revision}/${version_id}/coverage/coverage.${build_variant}.${task_name}
aws_key: ${AWS_ACCESS_KEY_ID}
aws_secret: ${AWS_SECRET_ACCESS_KEY}
aws_session_token: ${AWS_SESSION_TOKEN}
bucket: ${bucket_name}
local_file: src/.coverage
permissions: public-read
content_type: text/html
display_name: Raw Coverage Report
optional: "true"
type: setup
# Upload mo artifacts
upload mo artifacts:
- command: ec2.assume_role
params:
role_arn: ${assume_role_arn}
type: setup
- command: archive.targz_pack
params:
target: mongo-coredumps.tgz
source_dir: ./
include:
- ./**.core
- ./**.mdmp
- command: s3.put
params:
remote_file: ${build_variant}/${revision}/${version_id}/${build_id}/coredumps/${task_id}-${execution}-mongodb-coredumps.tar.gz
aws_key: ${AWS_ACCESS_KEY_ID}
aws_secret: ${AWS_SECRET_ACCESS_KEY}
aws_session_token: ${AWS_SESSION_TOKEN}
bucket: ${bucket_name}
local_file: mongo-coredumps.tgz
permissions: public-read
content_type: ${content_type|application/x-gzip}
display_name: Core Dumps - Execution
optional: "true"
type: setup
- command: s3.put
params:
remote_file: ${build_variant}/${revision}/${version_id}/${build_id}/logs/${task_id}-${execution}-drivers-tools-logs.tar.gz
aws_key: ${AWS_ACCESS_KEY_ID}
aws_secret: ${AWS_SECRET_ACCESS_KEY}
aws_session_token: ${AWS_SESSION_TOKEN}
bucket: ${bucket_name}
local_file: ${DRIVERS_TOOLS}/.evergreen/test_logs.tar.gz
permissions: public-read
content_type: ${content_type|application/x-gzip}
display_name: drivers-tools-logs.tar.gz
optional: "true"
type: setup
# Upload test results
upload test results:
- command: attach.results
params:
file_location: ${DRIVERS_TOOLS}/results.json
- command: attach.xunit_results
params:
file: src/xunit-results/TEST-*.xml

File diff suppressed because it is too large Load Diff

View File

@ -1,680 +0,0 @@
buildvariants:
# Alternative hosts tests
- name: other-hosts-rhel9-fips-latest
tasks:
- name: .test-no-toolchain
display_name: Other hosts RHEL9-FIPS latest
run_on:
- rhel92-fips
batchtime: 1440
expansions:
VERSION: latest
NO_EXT: "1"
REQUIRE_FIPS: "1"
UV_PYTHON: /usr/bin/python3.11
tags: []
- name: other-hosts-rhel8-zseries-latest
tasks:
- name: .test-no-toolchain
display_name: Other hosts RHEL8-zseries latest
run_on:
- rhel8-zseries-small
batchtime: 1440
expansions:
VERSION: latest
NO_EXT: "1"
tags: []
- name: other-hosts-rhel8-power8-latest
tasks:
- name: .test-no-toolchain
display_name: Other hosts RHEL8-POWER8 latest
run_on:
- rhel8-power-small
batchtime: 1440
expansions:
VERSION: latest
NO_EXT: "1"
tags: []
- name: other-hosts-rhel8-arm64-latest
tasks:
- name: .test-no-toolchain
display_name: Other hosts RHEL8-arm64 latest
run_on:
- rhel82-arm64-small
batchtime: 1440
expansions:
VERSION: latest
NO_EXT: "1"
tags: []
- name: other-hosts-amazon2023-latest
tasks:
- name: .test-no-toolchain
display_name: Other hosts Amazon2023 latest
run_on:
- amazon2023-arm64-latest-large-m8g
batchtime: 1440
expansions:
VERSION: latest
NO_EXT: "1"
tags: [pr]
# Atlas connect tests
- name: atlas-connect-rhel8
tasks:
- name: .test-no-orchestration
display_name: Atlas connect RHEL8
run_on:
- rhel87-small
expansions:
TEST_NAME: atlas_connect
tags: [pr]
# Aws auth tests
- name: auth-aws-rhel8
tasks:
- name: .auth-aws
display_name: Auth AWS RHEL8
run_on:
- rhel87-small
tags: []
- name: auth-aws-win64
tasks:
- name: .auth-aws
display_name: Auth AWS Win64
run_on:
- windows-2022-latest-small
tags: []
- name: auth-aws-macos
tasks:
- name: .auth-aws !.auth-aws-web-identity !.auth-aws-ec2
display_name: Auth AWS macOS
run_on:
- macos-14
tags: [pr]
- name: auth-aws-ecs-macos
tasks:
- name: .auth-aws-ecs
display_name: Auth AWS ECS macOS
run_on:
- ubuntu2404-small
tags: [pr]
# Aws lambda tests
- name: faas-lambda
tasks:
- name: .aws_lambda
display_name: FaaS Lambda
run_on:
- rhel87-small
# Backport pr tests
- name: backport-pr
tasks:
- name: backport-pr
display_name: Backport PR
run_on:
- rhel87-small
# Compression tests
- name: compression-snappy-rhel8
tasks:
- name: .test-standard
display_name: Compression snappy RHEL8
run_on:
- rhel87-small
expansions:
COMPRESSOR: snappy
- name: compression-zlib-rhel8
tasks:
- name: .test-standard
display_name: Compression zlib RHEL8
run_on:
- rhel87-small
expansions:
COMPRESSOR: zlib
- name: compression-zstd-rhel8
tasks:
- name: .test-standard !.server-4.2
display_name: Compression zstd RHEL8
run_on:
- rhel87-small
expansions:
COMPRESSOR: zstd
- name: compression-zstd-ubuntu-22
tasks:
- name: .test-standard !.server-4.2 !.server-4.4 !.server-5.0 .python-3.14
- name: .test-standard !.server-4.2 !.server-4.4 !.server-5.0 .python-3.14t
display_name: Compression zstd Ubuntu-22
run_on:
- ubuntu2204-small
expansions:
COMPRESSOR: ztsd
# Coverage report tests
- name: coverage-report
tasks:
- name: coverage-report
display_name: Coverage Report
run_on:
- rhel87-small
# Disable test commands tests
- name: disable-test-commands-rhel8
tasks:
- name: .test-standard .server-latest
display_name: Disable test commands RHEL8
run_on:
- rhel87-small
expansions:
AUTH: auth
SSL: ssl
DISABLE_TEST_COMMANDS: "1"
# Doctests tests
- name: doctests-rhel8
tasks:
- name: .test-non-standard .standalone-noauth-nossl
display_name: Doctests RHEL8
run_on:
- rhel87-small
expansions:
TEST_NAME: doctest
# Encryption tests
- name: encryption-rhel8
tasks:
- name: .test-non-standard
display_name: Encryption RHEL8
run_on:
- rhel87-small
batchtime: 1440
expansions:
TEST_NAME: encryption
tags: [encryption_tag]
- name: encryption-macos
tasks:
- name: .test-non-standard !.pypy
display_name: Encryption macOS
run_on:
- macos-14
batchtime: 1440
expansions:
TEST_NAME: encryption
tags: [encryption_tag]
- name: encryption-win64
tasks:
- name: .test-non-standard !.pypy
display_name: Encryption Win64
run_on:
- windows-2022-latest-small
batchtime: 1440
expansions:
TEST_NAME: encryption
tags: [encryption_tag]
- name: encryption-crypt_shared-rhel8
tasks:
- name: .test-non-standard
display_name: Encryption crypt_shared RHEL8
run_on:
- rhel87-small
batchtime: 1440
expansions:
TEST_NAME: encryption
TEST_CRYPT_SHARED: "true"
tags: [encryption_tag]
- name: encryption-crypt_shared-macos
tasks:
- name: .test-non-standard !.pypy
display_name: Encryption crypt_shared macOS
run_on:
- macos-14
batchtime: 1440
expansions:
TEST_NAME: encryption
TEST_CRYPT_SHARED: "true"
tags: [encryption_tag]
- name: encryption-crypt_shared-win64
tasks:
- name: .test-non-standard !.pypy
display_name: Encryption crypt_shared Win64
run_on:
- windows-2022-latest-small
batchtime: 1440
expansions:
TEST_NAME: encryption
TEST_CRYPT_SHARED: "true"
tags: [encryption_tag]
- name: encryption-pyopenssl-rhel8
tasks:
- name: .test-non-standard
display_name: Encryption PyOpenSSL RHEL8
run_on:
- rhel87-small
batchtime: 1440
expansions:
TEST_NAME: encryption
SUB_TEST_NAME: pyopenssl
tags: [encryption_tag]
# Enterprise auth tests
- name: auth-enterprise-rhel8
tasks:
- name: .test-standard-auth .auth !.free-threaded
display_name: Auth Enterprise RHEL8
run_on:
- rhel87-small
expansions:
TEST_NAME: enterprise_auth
AUTH: auth
- name: auth-enterprise-macos
tasks:
- name: .test-standard-auth !.pypy .auth !.free-threaded
display_name: Auth Enterprise macOS
run_on:
- macos-14
expansions:
TEST_NAME: enterprise_auth
AUTH: auth
- name: auth-enterprise-win64
tasks:
- name: .test-standard-auth !.pypy .auth !.free-threaded
display_name: Auth Enterprise Win64
run_on:
- windows-2022-latest-small
expansions:
TEST_NAME: enterprise_auth
AUTH: auth
# Green framework tests
- name: green-gevent-rhel8
tasks:
- name: .test-standard .sync !.free-threaded
display_name: Green Gevent RHEL8
run_on:
- rhel87-small
expansions:
GREEN_FRAMEWORK: gevent
# Import time tests
- name: import-time
tasks:
- name: check-import-time
display_name: Import Time
run_on:
- rhel87-small
# Kms tests
- name: kms
tasks:
- name: test-gcpkms
batchtime: 1440
- name: test-gcpkms-fail
- name: test-azurekms
batchtime: 1440
- name: test-azurekms-fail
display_name: KMS
run_on:
- debian11-small
# Load balancer tests
- name: load-balancer
tasks:
- name: .test-non-standard .server-6.0 .sharded_cluster-auth-ssl
- name: .test-non-standard .server-7.0 .sharded_cluster-auth-ssl
- name: .test-non-standard .server-8.0 .sharded_cluster-auth-ssl
- name: .test-non-standard .server-rapid .sharded_cluster-auth-ssl
- name: .test-non-standard .server-latest .sharded_cluster-auth-ssl
display_name: Load Balancer
run_on:
- rhel87-small
batchtime: 1440
expansions:
TEST_NAME: load_balancer
# Min support tests
- name: min-support-rhel8
tasks:
- name: .test-min-support
display_name: Min Support RHEL8
run_on:
- rhel87-small
# Mockupdb tests
- name: mockupdb-rhel8
tasks:
- name: .test-no-orchestration
display_name: MockupDB RHEL8
run_on:
- rhel87-small
expansions:
TEST_NAME: mockupdb
tags: [pr]
# Mod wsgi tests
- name: mod_wsgi-ubuntu-22
tasks:
- name: .mod_wsgi
display_name: Mod_WSGI Ubuntu-22
run_on:
- ubuntu2204-small
expansions:
MOD_WSGI_VERSION: "4"
# No c ext tests
- name: no-c-ext-rhel8
tasks:
- name: .test-standard
display_name: No C Ext RHEL8
run_on:
- rhel87-small
expansions:
NO_EXT: "1"
# No server tests
- name: no-server-rhel8
tasks:
- name: .test-no-orchestration
display_name: No server RHEL8
run_on:
- rhel87-small
tags: [pr]
# Ocsp tests
- name: ocsp-rhel8
tasks:
- name: .ocsp
display_name: OCSP RHEL8
run_on:
- rhel87-small
batchtime: 10080
- name: ocsp-win64
tasks:
- name: .ocsp-rsa !.ocsp-staple .latest
- name: .ocsp-rsa !.ocsp-staple .4.4
display_name: OCSP Win64
run_on:
- windows-2022-latest-small
batchtime: 10080
- name: ocsp-macos
tasks:
- name: .ocsp-rsa !.ocsp-staple .latest
- name: .ocsp-rsa !.ocsp-staple .4.4
display_name: OCSP macOS
run_on:
- macos-14
batchtime: 10080
# Oidc auth tests
- name: auth-oidc-ubuntu-22
tasks:
- name: .auth_oidc_remote
display_name: Auth OIDC Ubuntu-22
run_on:
- ubuntu2204-small
batchtime: 1440
- name: auth-oidc-local-ubuntu-22
tasks:
- name: "!.auth_oidc_remote .auth_oidc"
display_name: Auth OIDC Local Ubuntu-22
run_on:
- ubuntu2204-small
batchtime: 1440
expansions:
COVERAGE: "1"
tags: [pr]
- name: auth-oidc-macos
tasks:
- name: "!.auth_oidc_remote .auth_oidc"
display_name: Auth OIDC macOS
run_on:
- macos-14
batchtime: 1440
- name: auth-oidc-win64
tasks:
- name: "!.auth_oidc_remote .auth_oidc"
display_name: Auth OIDC Win64
run_on:
- windows-2022-latest-small
batchtime: 1440
# Perf tests
- name: performance-benchmarks
tasks:
- name: .perf
display_name: Performance Benchmarks
run_on:
- rhel90-dbx-perf-large
batchtime: 1440
# Pyopenssl tests
- name: pyopenssl-rhel8
tasks:
- name: .test-standard .sync
- name: .test-standard .async .replica_set-noauth-ssl
display_name: PyOpenSSL RHEL8
run_on:
- rhel87-small
batchtime: 1440
expansions:
SUB_TEST_NAME: pyopenssl
- name: pyopenssl-macos
tasks:
- name: .test-standard !.pypy .sync
- name: .test-standard !.pypy .async .replica_set-noauth-ssl
display_name: PyOpenSSL macOS
run_on:
- rhel87-small
batchtime: 1440
expansions:
SUB_TEST_NAME: pyopenssl
- name: pyopenssl-win64
tasks:
- name: .test-standard !.pypy .sync
- name: .test-standard !.pypy .async .replica_set-noauth-ssl
display_name: PyOpenSSL Win64
run_on:
- windows-2022-latest-small
batchtime: 1440
expansions:
SUB_TEST_NAME: pyopenssl
# Search index tests
- name: search-index-helpers-rhel8
tasks:
- name: .search_index
display_name: Search Index Helpers RHEL8
run_on:
- rhel87-small
# Server version tests
- name: mongodb-v4.2
tasks:
- name: .server-version
display_name: "* MongoDB v4.2"
run_on:
- rhel87-small
expansions:
VERSION: "4.2"
tags: [coverage_tag]
- name: mongodb-v4.4
tasks:
- name: .server-version
display_name: "* MongoDB v4.4"
run_on:
- rhel87-small
expansions:
VERSION: "4.4"
tags: [coverage_tag]
- name: mongodb-v5.0
tasks:
- name: .server-version
display_name: "* MongoDB v5.0"
run_on:
- rhel87-small
expansions:
VERSION: "5.0"
tags: [coverage_tag]
- name: mongodb-v6.0
tasks:
- name: .server-version
display_name: "* MongoDB v6.0"
run_on:
- rhel87-small
expansions:
VERSION: "6.0"
tags: [coverage_tag]
- name: mongodb-v7.0
tasks:
- name: .server-version
display_name: "* MongoDB v7.0"
run_on:
- rhel87-small
expansions:
VERSION: "7.0"
tags: [coverage_tag]
- name: mongodb-v8.0
tasks:
- name: .server-version
display_name: "* MongoDB v8.0"
run_on:
- rhel87-small
expansions:
VERSION: "8.0"
tags: [coverage_tag]
- name: mongodb-rapid
tasks:
- name: .server-version
display_name: "* MongoDB rapid"
run_on:
- rhel87-small
expansions:
VERSION: rapid
tags: [coverage_tag]
- name: mongodb-latest
tasks:
- name: .server-version
display_name: "* MongoDB latest"
run_on:
- rhel87-small
expansions:
VERSION: latest
tags: [coverage_tag]
# Stable api tests
- name: stable-api-require-v1-rhel8-auth
tasks:
- name: .test-standard !.replica_set-noauth-ssl .server-5.0
- name: .test-standard !.replica_set-noauth-ssl .server-6.0
- name: .test-standard !.replica_set-noauth-ssl .server-7.0
- name: .test-standard !.replica_set-noauth-ssl .server-8.0
- name: .test-standard !.replica_set-noauth-ssl .server-rapid
- name: .test-standard !.replica_set-noauth-ssl .server-latest
display_name: Stable API require v1 RHEL8 Auth
run_on:
- rhel87-small
expansions:
AUTH: auth
REQUIRE_API_VERSION: "1"
MONGODB_API_VERSION: "1"
tags: [versionedApi_tag]
- name: stable-api-accept-v2-rhel8-auth
tasks:
- name: .test-standard .server-5.0 .standalone-noauth-nossl
- name: .test-standard .server-6.0 .standalone-noauth-nossl
- name: .test-standard .server-7.0 .standalone-noauth-nossl
- name: .test-standard .server-8.0 .standalone-noauth-nossl
- name: .test-standard .server-rapid .standalone-noauth-nossl
- name: .test-standard .server-latest .standalone-noauth-nossl
display_name: Stable API accept v2 RHEL8 Auth
run_on:
- rhel87-small
expansions:
AUTH: auth
ORCHESTRATION_FILE: versioned-api-testing.json
tags: [versionedApi_tag]
# Standard nonlinux tests
- name: test-macos
tasks:
- name: .test-standard !.pypy
display_name: "* Test macOS"
run_on:
- macos-14
tags: [standard-non-linux]
- name: test-macos-arm64
tasks:
- name: .test-standard !.pypy .server-6.0
- name: .test-standard !.pypy .server-7.0
- name: .test-standard !.pypy .server-8.0
- name: .test-standard !.pypy .server-rapid
- name: .test-standard !.pypy .server-latest
display_name: "* Test macOS Arm64"
run_on:
- macos-14-arm64
tags: [standard-non-linux]
- name: test-win64
tasks:
- name: .test-standard !.pypy
- name: .test-no-orchestration !.pypy
display_name: "* Test Win64"
run_on:
- windows-2022-latest-small
tags: [standard-non-linux]
- name: test-win32
tasks:
- name: .test-standard !.pypy
display_name: "* Test Win32"
run_on:
- windows-64-vsMulti-small
expansions:
IS_WIN32: "1"
tags: [standard-non-linux]
# Storage engine tests
- name: storage-inmemory-rhel8
tasks:
- name: .test-standard .standalone-noauth-nossl
display_name: Storage InMemory RHEL8
run_on:
- rhel87-small
expansions:
STORAGE_ENGINE: inmemory
# Test numpy tests
- name: test-numpy-rhel8
tasks:
- name: .test-numpy
display_name: Test Numpy RHEL8
run_on:
- rhel87-small
tags: [binary, vector, pr]
- name: test-numpy-macos
tasks:
- name: .test-numpy
display_name: Test Numpy macOS
run_on:
- macos-14
tags: [binary, vector]
- name: test-numpy-macos-arm64
tasks:
- name: .test-numpy
display_name: Test Numpy macOS Arm64
run_on:
- macos-14-arm64
tags: [binary, vector]
- name: test-numpy-win64
tasks:
- name: .test-numpy
display_name: Test Numpy Win64
run_on:
- windows-2022-latest-small
tags: [binary, vector]
- name: test-numpy-win32
tasks:
- name: .test-numpy
display_name: Test Numpy Win32
run_on:
- windows-64-vsMulti-small
expansions:
IS_WIN32: "1"
tags: [binary, vector]

View File

@ -0,0 +1,19 @@
#!/bin/sh
set -o xtrace # Write all commands first to stderr
set -o errexit # Exit the script with error if any of the commands fail
# Copy PyMongo's test certificates over driver-evergreen-tools'
cp ${PROJECT_DIRECTORY}/test/certificates/* ${DRIVERS_TOOLS}/.evergreen/x509gen/
# Replace MongoOrchestration's client certificate.
cp ${PROJECT_DIRECTORY}/test/certificates/client.pem ${MONGO_ORCHESTRATION_HOME}/lib/client.pem
if [ -w /etc/hosts ]; then
SUDO=""
else
SUDO="sudo"
fi
# Add 'server' and 'hostname_not_in_cert' as a hostnames
echo "127.0.0.1 server" | $SUDO tee -a /etc/hosts
echo "127.0.0.1 hostname_not_in_cert" | $SUDO tee -a /etc/hosts

View File

@ -1,5 +0,0 @@
#!/bin/bash
set -eu
. .evergreen/scripts/setup-dev-env.sh
just "$@"

View File

@ -1,56 +0,0 @@
#!/bin/bash
PYMONGO=$(dirname "$(cd "$(dirname "$0")" || exit; pwd)")
rm $PYMONGO/test/transactions/legacy/errors-client.json # PYTHON-1894
rm $PYMONGO/test/connection_monitoring/wait-queue-fairness.json # PYTHON-1873
rm $PYMONGO/test/discovery_and_monitoring/unified/pool-clear-application-error.json # PYTHON-4918
rm $PYMONGO/test/discovery_and_monitoring/unified/pool-clear-checkout-error.json # PYTHON-4918
rm $PYMONGO/test/discovery_and_monitoring/unified/pool-clear-min-pool-size-error.json # PYTHON-4918
rm $PYMONGO/test/client-side-encryption/spec/unified/client-bulkWrite-qe.json # PYTHON-4929
# Python doesn't implement DRIVERS-3064
rm $PYMONGO/test/collection_management/listCollections-rawdata.json
rm $PYMONGO/test/crud/unified/aggregate-rawdata.json
rm $PYMONGO/test/crud/unified/bulkWrite-deleteMany-rawdata.json
rm $PYMONGO/test/crud/unified/bulkWrite-deleteOne-rawdata.json
rm $PYMONGO/test/crud/unified/bulkWrite-replaceOne-rawdata.json
rm $PYMONGO/test/crud/unified/bulkWrite-updateMany-rawdata.json
rm $PYMONGO/test/crud/unified/bulkWrite-updateOne-rawdata.json
rm $PYMONGO/test/crud/unified/client-bulkWrite-delete-rawdata.json
rm $PYMONGO/test/crud/unified/client-bulkWrite-replaceOne-rawdata.json
rm $PYMONGO/test/crud/unified/client-bulkWrite-update-rawdata.json
rm $PYMONGO/test/crud/unified/count-rawdata.json
rm $PYMONGO/test/crud/unified/countDocuments-rawdata.json
rm $PYMONGO/test/crud/unified/db-aggregate-rawdata.json
rm $PYMONGO/test/crud/unified/deleteMany-rawdata.json
rm $PYMONGO/test/crud/unified/deleteOne-rawdata.json
rm $PYMONGO/test/crud/unified/distinct-rawdata.json
rm $PYMONGO/test/crud/unified/estimatedDocumentCount-rawdata.json
rm $PYMONGO/test/crud/unified/find-rawdata.json
rm $PYMONGO/test/crud/unified/findOneAndDelete-rawdata.json
rm $PYMONGO/test/crud/unified/findOneAndReplace-rawdata.json
rm $PYMONGO/test/crud/unified/findOneAndUpdate-rawdata.json
rm $PYMONGO/test/crud/unified/insertMany-rawdata.json
rm $PYMONGO/test/crud/unified/insertOne-rawdata.json
rm $PYMONGO/test/crud/unified/replaceOne-rawdata.json
rm $PYMONGO/test/crud/unified/updateMany-rawdata.json
rm $PYMONGO/test/crud/unified/updateOne-rawdata.json
rm $PYMONGO/test/index_management/index-rawdata.json
# PyMongo does not support modifyCollection
rm $PYMONGO/test/collection_management/modifyCollection-*.json
# PYTHON-5248 - Remove support for MongoDB 4.0
find /$PYMONGO/test -type f -name 'pre-42-*.json' -delete
# PYTHON-3359 - Remove Database and Collection level timeout override
rm $PYMONGO/test/csot/override-collection-timeoutMS.json
rm $PYMONGO/test/csot/override-database-timeoutMS.json
# PYTHON-2943 - Socks5 Proxy Support
rm $PYMONGO/test/uri_options/proxy-options.json
# PYTHON-5517 - Avoid clearing the connection pool when the server connection rate limiter triggers
rm $PYMONGO/test/discovery_and_monitoring/unified/backpressure-*.json
echo "Done removing unimplemented tests"

View File

@ -1,207 +0,0 @@
#!/bin/bash
# Resync test files from the specifications repo.
set -eu
PYMONGO=$(dirname "$(cd "$(dirname "$0")"; pwd)")
SPECS=${MDB_SPECS:-~/Work/specifications}
help (){
echo "Usage: resync_specs.sh [-bcsp] spec"
echo "Required arguments:"
echo " spec determines which folder the spec tests will be copied from."
echo "Optional flags:"
echo " -b is used to add a string to the blocklist for that next run. Can be used"
echo " any number of times on a single command to block multiple patterns."
echo " You can use any regex pattern (it is passed to 'grep -Ev')."
echo " -c is used to set a branch or commit that will be checked out in the"
echo " specifications repo before copying."
echo " -s is used to set a unique path to the specs repo for that specific"
echo " run."
echo "Notes:"
echo "You can export the environment variable MDB_SPECS to set the specs"
echo " repo similar to -s, but this will persist between runs until you "
echo "unset it."
}
# Parse flag args
BRANCH=''
BLOCKLIST='.*\.yml'
while getopts 'b:c:s:' flag; do
case "${flag}" in
b) BLOCKLIST+="|$OPTARG"
;;
c) BRANCH="${OPTARG}"
;;
s) SPECS="${OPTARG}"
;;
*) help; exit 0
;;
esac
done
shift $((OPTIND-1))
if [ -n "$BRANCH" ]
then
git -C $SPECS checkout $BRANCH
fi
# Ensure the JSON files are up to date.
if ! [ -n "${CI:-}" ]
then
cd $SPECS/source
make
cd -
fi
# cpjson unified-test-format/tests/invalid unified-test-format/invalid
# * param1: Path to spec tests dir in specifications repo
# * param2: Path to where the corresponding tests live in Python.
cpjson () {
find "$PYMONGO"/test/$2 -type f -delete
cd "$SPECS"/source/$1
find . -name '*.json' | grep -Ev "${BLOCKLIST}" | cpio -pdm \
$PYMONGO/test/$2
printf "\nIgnored files for ${PWD}:\n"
IGNORED_FILES="$(printf "\n%s\n" "$(diff <(find . -name '*.json' | sort) \
<(find . -name '*.json' | grep -Ev "${BLOCKLIST}" | sort))" | \
sed -e '/^[0-9]/d' | sed -e 's|< ./||g' )"
printf "%s\n" $IGNORED_FILES
cd "$PYMONGO"/test/$2
printf "%s\n" $IGNORED_FILES | xargs git checkout master
}
for spec in "$@"
do
# Match the spec dir name, the python test dir name, and/or common abbreviations.
case "$spec" in
auth)
cpjson auth/tests/ auth
;;
bson-binary-vector|bson_binary_vector)
cpjson bson-binary-vector/tests/ bson_binary_vector
;;
bson-corpus|bson_corpus)
cpjson bson-corpus/tests/ bson_corpus
;;
max-staleness|max_staleness)
cpjson max-staleness/tests/ max_staleness
;;
collection-management|collection_management)
cpjson collection-management/tests/ collection_management
;;
connection-string|connection_string)
cpjson connection-string/tests/ connection_string/test
;;
change-streams|change_streams)
cpjson change-streams/tests/ change_streams/
;;
client-backpressure|client_backpressure)
cpjson client-backpressure/tests client-backpressure
;;
client-side-encryption|csfle|fle)
cpjson client-side-encryption/tests/ client-side-encryption/spec
cpjson client-side-encryption/corpus/ client-side-encryption/corpus
cpjson client-side-encryption/external/ client-side-encryption/external
cpjson client-side-encryption/limits/ client-side-encryption/limits
cpjson client-side-encryption/etc/data client-side-encryption/etc/data
;;
connection-monitoring|connection_monitoring)
cpjson connection-monitoring-and-pooling/tests/cmap-format connection_monitoring
;;
connection-logging|connection_logging)
cpjson connection-monitoring-and-pooling/tests/logging connection_logging
;;
cmap|CMAP|connection-monitoring-and-pooling)
cpjson connection-monitoring-and-pooling/tests/logging connection_logging
cpjson connection-monitoring-and-pooling/tests/cmap-format connection_monitoring
;;
apm|APM|command-monitoring|command_monitoring)
cpjson command-logging-and-monitoring/tests/monitoring command_monitoring
;;
command-logging|command_logging)
cpjson command-logging-and-monitoring/tests/logging command_logging
;;
clam|CLAM|command-logging-and-monitoring|command_logging_and_monitoring)
cpjson command-logging-and-monitoring/tests/logging command_logging
cpjson command-logging-and-monitoring/tests/monitoring command_monitoring
;;
crud|CRUD)
cpjson crud/tests/ crud
;;
csot|CSOT|client-side-operations-timeout)
cpjson client-side-operations-timeout/tests csot
;;
gridfs)
cpjson gridfs/tests gridfs
;;
handshake)
cpjson mongodb-handshake/tests handshake
;;
index|index-management)
cpjson index-management/tests index_management
;;
load-balancers|load_balancer)
cpjson load-balancers/tests load_balancer
;;
srv|SRV|initial-dns-seedlist-discovery|srv_seedlist)
cpjson initial-dns-seedlist-discovery/tests/ srv_seedlist
;;
read-write-concern|read_write_concern)
cpjson read-write-concern/tests/operation read_write_concern/operation
;;
retryable-reads|retryable_reads)
cpjson retryable-reads/tests/ retryable_reads
;;
retryable-writes|retryable_writes)
cpjson retryable-writes/tests/ retryable_writes
;;
run-command|run_command)
cpjson run-command/tests/ run_command
;;
sdam|SDAM|server-discovery-and-monitoring|discovery_and_monitoring)
cpjson server-discovery-and-monitoring/tests/errors \
discovery_and_monitoring/errors
cpjson server-discovery-and-monitoring/tests/rs \
discovery_and_monitoring/rs
cpjson server-discovery-and-monitoring/tests/sharded \
discovery_and_monitoring/sharded
cpjson server-discovery-and-monitoring/tests/single \
discovery_and_monitoring/single
cpjson server-discovery-and-monitoring/tests/unified \
discovery_and_monitoring/unified
cpjson server-discovery-and-monitoring/tests/load-balanced \
discovery_and_monitoring/load-balanced
;;
sdam-monitoring|sdam_monitoring)
cpjson server-discovery-and-monitoring/tests/monitoring sdam_monitoring
;;
server-selection|server_selection)
cpjson server-selection/tests/ server_selection
rm -rf $PYMONGO/test/server_selection/logging # these tests live in server_selection_logging
cpjson server-selection/tests/logging server_selection_logging
;;
server-selection-logging|server_selection_logging)
cpjson server-selection/tests/logging server_selection_logging
;;
sessions)
cpjson sessions/tests/ sessions
;;
transactions|transactions-convenient-api)
cpjson transactions/tests/ transactions
cpjson transactions-convenient-api/tests/ transactions-convenient-api
;;
unified|unified-test-format)
cpjson unified-test-format/tests/ unified-test-format/
;;
uri|uri-options|uri_options)
cpjson uri-options/tests uri_options
cp "$SPECS"/source/uri-options/tests/*.pem $PYMONGO/test/uri_options
;;
stable-api|versioned-api)
cpjson versioned-api/tests versioned-api
;;
*)
echo "Do not know how to resync spec tests for '${spec}'"
help
;;
esac
done

View File

@ -0,0 +1,47 @@
#!/bin/bash
# Don't trace to avoid secrets showing up in the logs
set -o errexit
echo "Running enterprise authentication tests"
export JAVA_HOME=/opt/java/jdk8
PYTHON_VERSION=$(${PYTHON_BINARY} -c 'import sys; sys.stdout.write(str(sys.version_info[0]))')
PLATFORM="$(${PYTHON_BINARY} -c 'import platform, sys; sys.stdout.write(platform.system())')"
export DB_USER="bob"
export DB_PASSWORD="pwd123"
# There is no kerberos package for Jython, but we do want to test PLAIN.
if [ ${PLATFORM} != "Java" ]; then
# PyMongo 2.x doesn't support GSSAPI on Windows.
if [ "Windows_NT" != "$OS" ]; then
echo "Writing keytab"
echo ${KEYTAB_BASE64} | base64 -d > ${PROJECT_DIRECTORY}/.evergreen/drivers.keytab
echo "Running kinit"
kinit -k -t ${PROJECT_DIRECTORY}/.evergreen/drivers.keytab -p ${PRINCIPAL}
echo "Setting GSSAPI variables"
export GSSAPI_HOST=${SASL_HOST}
export GSSAPI_PORT=${SASL_PORT}
export GSSAPI_PRINCIPAL=${PRINCIPAL}
fi
EXTRA_ARGS=""
else
# Keep Jython 2.5 from running out of memory.
EXTRA_ARGS="-J-XX:-UseGCOverheadLimit -J-Xmx4096m"
fi
# Set verbose test output flag.
if [ "$PYTHON_VERSION" = "3" ]; then
# With Python 3, the tests do not accept a "--verbosity=2" flag.
TEST_VERBOSITY="-v"
else
# With Python 2, the tests accepts a "-v" flag but only "--verbosity=2"
# causes the verbose output we want.
TEST_VERBOSITY="--verbosity=2"
fi
echo "Running tests"
${PYTHON_BINARY} setup.py clean
${PYTHON_BINARY} $EXTRA_ARGS setup.py nosetests $TEST_VERBOSITY

View File

@ -0,0 +1,47 @@
#!/bin/sh
set -o xtrace
set -o errexit
APACHE=$(command -v apache2 || command -v /usr/lib/apache2/mpm-prefork/apache2) || true
if [ -z "$APACHE" ]; then
echo "Could not find apache2 binary"
exit 1
fi
PYTHON_VERSION=$(${PYTHON_BINARY} -c "import sys; sys.stdout.write('.'.join(str(val) for val in sys.version_info[:2]))")
if [ $MOD_WSGI_VERSION = "2.8" ] && [ $PYTHON_VERSION = "2.7" ]; then
# mod_wsgi 2.8 segfaults when built against the toolchain Python 2.7. Build
# against the system Python 2.7 instead.
git clone https://github.com/GrahamDumpleton/mod_wsgi.git
cd mod_wsgi
git checkout tags/2.8
./configure
make
export MOD_WSGI_SO=$(pwd)/.libs/mod_wsgi.so
cd ..
else
export MOD_WSGI_SO=/opt/python/mod_wsgi/python_version/$PYTHON_VERSION/mod_wsgi_version/$MOD_WSGI_VERSION/mod_wsgi.so
export PYTHONHOME=/opt/python/$PYTHON_VERSION
fi
cd ..
$APACHE -k start -f ${PROJECT_DIRECTORY}/test/mod_wsgi_test/apache22ubuntu1204.conf
trap "$APACHE -k stop -f ${PROJECT_DIRECTORY}/test/mod_wsgi_test/apache22ubuntu1204.conf" EXIT HUP
set +e
wget -t 1 -T 10 -O - "http://localhost:8080${PROJECT_DIRECTORY}"
STATUS=$?
set -e
# Debug
cat error_log
if [ $STATUS != 0 ]; then
exit $STATUS
fi
${PYTHON_BINARY} ${PROJECT_DIRECTORY}/test/mod_wsgi_test/test_client.py -n 25000 -t 100 parallel http://localhost:8080${PROJECT_DIRECTORY}
${PYTHON_BINARY} ${PROJECT_DIRECTORY}/test/mod_wsgi_test/test_client.py -n 25000 serial http://localhost:8080${PROJECT_DIRECTORY}

View File

@ -1,32 +0,0 @@
#!/bin/bash
# Script run on an ECS host to test MONGODB-AWS.
set -eu
############################################
# Main Program #
############################################
if [[ -z "$1" ]]; then
echo "usage: $0 <MONGODB_URI>"
exit 1
fi
export MONGODB_URI="$1"
if echo "$MONGODB_URI" | grep -q "@"; then
echo "MONGODB_URI unexpectedly contains user credentials in ECS test!";
exit 1
fi
# Now we can safely enable xtrace
set -o xtrace
# Install a c compiler.
apt-get -qq update < /dev/null > /dev/null
apt-get -q install -y build-essential
export SET_XTRACE_ON=1
cd src
rm -rf .venv
rm -f .evergreen/scripts/test-env.sh || true
rm -f .evergreen/scripts/env.sh || true
bash ./.evergreen/just.sh setup-tests auth_aws ecs-remote
bash .evergreen/just.sh run-tests

View File

@ -1,17 +0,0 @@
#!/bin/bash
# Script run on a remote host to test MONGODB-OIDC.
set -eu
echo "Running MONGODB-OIDC authentication tests on ${OIDC_ENV}..."
if [ ${OIDC_ENV} == "k8s" ]; then
SUB_TEST_NAME=$K8S_VARIANT-remote
else
SUB_TEST_NAME=$OIDC_ENV-remote
sudo apt-get install -y python3-dev build-essential
fi
bash ./.evergreen/just.sh setup-tests auth_oidc $SUB_TEST_NAME
bash ./.evergreen/just.sh run-tests "${@:1}"
echo "Running MONGODB-OIDC authentication tests on ${OIDC_ENV}... done."

View File

@ -1,45 +1,78 @@
#!/bin/bash #!/bin/sh
# Run a test suite that was configured with setup-tests.sh. set -o xtrace # Write all commands first to stderr
set -eu set -o errexit # Exit the script with error if any of the commands fail
SCRIPT_DIR=$(dirname ${BASH_SOURCE:-$0}) # Supported/used environment variables:
SCRIPT_DIR="$( cd -- "$SCRIPT_DIR" > /dev/null 2>&1 && pwd )" # AUTH Set to enable authentication. Defaults to "noauth"
ROOT_DIR="$(dirname $SCRIPT_DIR)" # SSL Set to enable SSL. Defaults to "nossl"
# PYTHON_BINARY The Python version to use. Defaults to whatever is available
# C_EXTENSIONS Pass --no_ext to setup.py, or not.
PREV_DIR=$(pwd)
cd $ROOT_DIR
# Try to source the env file. AUTH=${AUTH:-noauth}
if [ -f $SCRIPT_DIR/scripts/env.sh ]; then SSL=${SSL:-nossl}
echo "Sourcing env inputs" PYTHON_BINARY=${PYTHON_BINARY:-}
. $SCRIPT_DIR/scripts/env.sh C_EXTENSIONS=${C_EXTENSIONS:-}
else
echo "Not sourcing env inputs" export JAVA_HOME=/opt/java/jdk8
if [ "$AUTH" != "noauth" ]; then
export DB_USER="bob"
export DB_PASSWORD="pwd123"
fi fi
# Handle test inputs. if [ -z "$PYTHON_BINARY" ]; then
if [ -f $SCRIPT_DIR/scripts/test-env.sh ]; then PYTHON=$(command -v python || command -v python3) || true
echo "Sourcing test inputs" if [ -z "$PYTHON" ]; then
. $SCRIPT_DIR/scripts/test-env.sh echo "Cannot test without python or python3 installed!"
exit 1
fi
else else
echo "Missing test inputs, please run 'just setup-tests'" PYTHON="$PYTHON_BINARY"
exit 1
fi fi
cleanup_tests() { PYTHON_VERSION=$($PYTHON -c 'import sys; sys.stdout.write(str(sys.version_info[0]))')
# Avoid leaving the lock file in a changed state when we change the resolution type. PLATFORM=$($PYTHON -c 'import platform, sys; sys.stdout.write(platform.system())')
if [ -n "${TEST_MIN_DEPS:-}" ]; then
git checkout uv.lock || true
fi
cd $PREV_DIR
}
trap "cleanup_tests" SIGINT ERR if [ "$SSL" = "ssl" ]; then
if [ "$PYTHON_VERSION" = "3" ]; then
# We cannot pass arguments to the "nosetests" command with Python 3
# because of the hacks in setup.py to work around nose/2to3 usage.
# Instead, use the "test" command directly to run only test/test_ssl.py.
# Unfortunately, this does not produce XML output.
TEST_CMD="test --test-suite test.test_ssl"
else
# With Python 2 we use the nosetests command
# Run only test/test_ssl.py and produces nosetests.xml output.
TEST_CMD="nosetests --tests test/test_ssl.py"
fi
else
# Run all the tests and produces nosetests.xml output.
TEST_CMD="nosetests"
fi
# Start the test runner. # Set verbose test output flag.
echo "Running tests with UV_PYTHON=${UV_PYTHON:-}..." if [ "$PYTHON_VERSION" = "3" ]; then
echo "UV_ARGS=${UV_ARGS}" # With Python 3, the tests do not accept a "--verbosity=2" flag.
uv run ${UV_ARGS} --reinstall-package pymongo .evergreen/scripts/run_tests.py "$@" TEST_VERBOSITY="-v"
echo "Running tests with UV_PYTHON=${UV_PYTHON:-}... done." else
# With Python 2, the tests accepts a "-v" flag but only "--verbosity=2"
# causes the verbose output we want.
TEST_VERBOSITY="--verbosity=2"
fi
cleanup_tests if [ "$PLATFORM" = "Java" ]; then
# Keep Jython 2.5 from running out of memory.
EXTRA_ARGS="-J-XX:-UseGCOverheadLimit -J-Xmx4096m"
else
EXTRA_ARGS=""
fi
echo "Running $AUTH tests over $SSL with python $PYTHON"
$PYTHON -c 'import sys; print(sys.version)'
# Run the tests, and store the results in Evergreen compatible XUnit XML
# files in the xunit-results/ directory.
$PYTHON setup.py clean
$PYTHON $EXTRA_ARGS setup.py $C_EXTENSIONS $TEST_CMD $TEST_VERBOSITY

View File

@ -1,41 +0,0 @@
#!/bin/bash
# Check for regressions in the import time of pymongo.
set -eu
HERE=$(dirname ${BASH_SOURCE:-$0})
source $HERE/env.sh
pushd $HERE/../.. >/dev/null
BASE_SHA="$1"
HEAD_SHA="$2"
# Set up the virtual env.
. $HERE/setup-dev-env.sh
uv venv --seed
source .venv/bin/activate
# Use the previous commit if this was not a PR run.
if [ "$BASE_SHA" == "$HEAD_SHA" ]; then
BASE_SHA=$(git rev-parse HEAD~1)
fi
function get_import_time() {
local log_file
python -m pip install -q ".[aws,encryption,gssapi,ocsp,snappy,zstd]"
# Import once to cache modules
python -c "import pymongo"
log_file="pymongo-$1.log"
python -X importtime -c "import pymongo" 2> $log_file
}
get_import_time $HEAD_SHA
git stash || true
git checkout $BASE_SHA
get_import_time $BASE_SHA
git checkout $HEAD_SHA
git stash apply || true
python tools/compare_import_time.py $HEAD_SHA $BASE_SHA
popd >/dev/null

View File

@ -1,14 +0,0 @@
#!/bin/bash
# Clean up resources at the end of an evergreen run.
set -eu
HERE=$(dirname ${BASH_SOURCE:-$0})
# Try to source the env file.
if [ -f $HERE/env.sh ]; then
echo "Sourcing env file"
source $HERE/env.sh
fi
rm -rf "${DRIVERS_TOOLS}" || true
rm -f $HERE/../../secrets-export.sh || true

View File

@ -1,114 +0,0 @@
#!/bin/bash
# Configure an evergreen test environment.
set -eu
# Get the current unique version of this checkout
# shellcheck disable=SC2154
if [ "${is_patch:-}" = "true" ]; then
# shellcheck disable=SC2154
CURRENT_VERSION="$(git describe)-patch-$version_id"
else
CURRENT_VERSION=latest
fi
PROJECT_DIRECTORY="$(pwd)"
DRIVERS_TOOLS="$(dirname $PROJECT_DIRECTORY)/drivers-tools"
CARGO_HOME=${CARGO_HOME:-${DRIVERS_TOOLS}/.cargo}
UV_TOOL_DIR=$PROJECT_DIRECTORY/.local/uv/tools
UV_CACHE_DIR=$PROJECT_DIRECTORY/.local/uv/cache
DRIVERS_TOOLS_BINARIES="$DRIVERS_TOOLS/.bin"
MONGODB_BINARIES="$DRIVERS_TOOLS/mongodb/bin"
# On Evergreen jobs, "CI" will be set, and we don't want to write to $HOME.
if [ "${CI:-}" == "true" ]; then
PYMONGO_BIN_DIR=${DRIVERS_TOOLS_BINARIES:-}
# We want to use a path that's already on PATH on spawn hosts.
else
PYMONGO_BIN_DIR=$HOME/cli_bin
fi
PATH_EXT="$MONGODB_BINARIES:$DRIVERS_TOOLS_BINARIES:$PYMONGO_BIN_DIR:\$PATH"
# Python has cygwin path problems on Windows. Detect prospective mongo-orchestration home directory
if [ "Windows_NT" = "${OS:-}" ]; then # Magic variable in cygwin
DRIVERS_TOOLS=$(cygpath -m $DRIVERS_TOOLS)
PROJECT_DIRECTORY=$(cygpath -m $PROJECT_DIRECTORY)
CARGO_HOME=$(cygpath -m $CARGO_HOME)
UV_TOOL_DIR=$(cygpath -m "$UV_TOOL_DIR")
UV_CACHE_DIR=$(cygpath -m "$UV_CACHE_DIR")
DRIVERS_TOOLS_BINARIES=$(cygpath -m "$DRIVERS_TOOLS_BINARIES")
MONGODB_BINARIES=$(cygpath -m "$MONGODB_BINARIES")
PYMONGO_BIN_DIR=$(cygpath -m "$PYMONGO_BIN_DIR")
fi
SCRIPT_DIR="$PROJECT_DIRECTORY/.evergreen/scripts"
if [ -f "$SCRIPT_DIR/env.sh" ]; then
echo "Reading $SCRIPT_DIR/env.sh file"
. "$SCRIPT_DIR/env.sh"
exit 0
fi
export MONGO_ORCHESTRATION_HOME="$DRIVERS_TOOLS/.evergreen/orchestration"
export MONGODB_BINARIES="$DRIVERS_TOOLS/mongodb/bin"
cat <<EOT > "$SCRIPT_DIR"/env.sh
export PROJECT_DIRECTORY="$PROJECT_DIRECTORY"
export CURRENT_VERSION="$CURRENT_VERSION"
export DRIVERS_TOOLS="$DRIVERS_TOOLS"
export MONGO_ORCHESTRATION_HOME="$MONGO_ORCHESTRATION_HOME"
export MONGODB_BINARIES="$MONGODB_BINARIES"
export DRIVERS_TOOLS_BINARIES="$DRIVERS_TOOLS_BINARIES"
export PROJECT_DIRECTORY="$PROJECT_DIRECTORY"
export CARGO_HOME="$CARGO_HOME"
export UV_TOOL_DIR="$UV_TOOL_DIR"
export UV_CACHE_DIR="$UV_CACHE_DIR"
export UV_TOOL_BIN_DIR="$DRIVERS_TOOLS_BINARIES"
export PYMONGO_BIN_DIR="$PYMONGO_BIN_DIR"
export PATH="$PATH_EXT"
# shellcheck disable=SC2154
export PROJECT="${project:-mongo-python-driver}"
export PIP_QUIET=1
EOT
# Write the .env file for drivers-tools.
rm -rf $DRIVERS_TOOLS
BRANCH=master
ORG=mongodb-labs
git clone --branch $BRANCH https://github.com/$ORG/drivers-evergreen-tools.git $DRIVERS_TOOLS
cat <<EOT > ${DRIVERS_TOOLS}/.env
SKIP_LEGACY_SHELL=1
DRIVERS_TOOLS="$DRIVERS_TOOLS"
MONGO_ORCHESTRATION_HOME="$MONGO_ORCHESTRATION_HOME"
MONGODB_BINARIES="$MONGODB_BINARIES"
EOT
# Add these expansions to make it easier to call out tests scripts from the EVG yaml
cat <<EOT > expansion.yml
DRIVERS_TOOLS: "$DRIVERS_TOOLS"
PROJECT_DIRECTORY: "$PROJECT_DIRECTORY"
EOT
# If the toolchain is available, symlink binaries to the bin dir. This has to be done
# after drivers-tools is cloned, since we might be using its binary dir.
_bin_path=""
if [ "Windows_NT" == "${OS:-}" ]; then
_bin_path="/cygdrive/c/Python/Current/Scripts"
elif [ "$(uname -s)" == "Darwin" ]; then
_bin_path="/Library/Frameworks/Python.Framework/Versions/Current/bin"
else
_bin_path="/opt/python/Current/bin"
fi
if [ -d "${_bin_path}" ]; then
_suffix=""
if [ "Windows_NT" == "${OS:-}" ]; then
_suffix=".exe"
fi
echo "Symlinking binaries from toolchain"
mkdir -p $PYMONGO_BIN_DIR
ln -s ${_bin_path}/just${_suffix} $PYMONGO_BIN_DIR/just${_suffix}
ln -s ${_bin_path}/uv${_suffix} $PYMONGO_BIN_DIR/uv${_suffix}
ln -s ${_bin_path}/uvx${_suffix} $PYMONGO_BIN_DIR/uvx${_suffix}
fi

View File

@ -1,50 +0,0 @@
#!/usr/bin/env bash
tools="$(realpath -s "../drivers-tools")"
pushd $tools/.evergreen/github_app || exit
owner="mongodb"
repo="mongo-python-driver"
# Bootstrap the app.
echo "bootstrapping"
source utils.sh
bootstrap drivers/comment-bot
# Run the app.
source ./secrets-export.sh
# Get a github access token for the git checkout.
echo "Getting github token..."
token=$(bash ./get-access-token.sh $repo $owner)
if [ -z "${token}" ]; then
echo "Failed to get github access token!"
popd || exit
exit 1
fi
echo "Getting github token... done."
popd || exit
# Make the git checkout and create a new branch.
echo "Creating the git checkout..."
branch="spec-resync-"$(date '+%m-%d-%Y')
git remote set-url origin https://x-access-token:${token}@github.com/$owner/$repo.git
git checkout -b $branch "origin/master"
git add ./test
git commit -am "resyncing specs $(date '+%m-%d-%Y')"
echo "Creating the git checkout... done."
git push origin $branch
resp=$(curl -L \
-X POST \
-H "Accept: application/vnd.github+json" \
-H "Authorization: Bearer $token" \
-H "X-GitHub-Api-Version: 2022-11-28" \
-d "{\"title\":\"[Spec Resync] $(date '+%m-%d-%Y')\",\"body\":\"$(cat "$1")\",\"head\":\"${branch}\",\"base\":\"master\"}" \
--url https://api.github.com/repos/$owner/$repo/pulls)
echo $resp | jq '.html_url'
echo "Creating the PR... done."
rm -rf $tools

View File

@ -1,4 +0,0 @@
#!/bin/bash
# Download all the task coverage files.
set -eu
aws s3 cp --recursive s3://"$1"/coverage/"$2"/"$3"/coverage/ coverage/

View File

@ -1,6 +0,0 @@
#!/bin/bash
# Entry point for the generate-config pre-commit hook.
set -eu
python .evergreen/scripts/generate_config.py

File diff suppressed because it is too large Load Diff

View File

@ -1,357 +0,0 @@
from __future__ import annotations
from dataclasses import dataclass
from inspect import getmembers, isfunction
from itertools import cycle, zip_longest
from pathlib import Path
from typing import Any
from shrub.v3.evg_build_variant import BuildVariant
from shrub.v3.evg_command import (
EvgCommandType,
ec2_assume_role,
s3_put,
subprocess_exec,
)
from shrub.v3.evg_project import EvgProject
from shrub.v3.evg_task import EvgTaskRef
from shrub.v3.shrub_service import ShrubService
##############
# Globals
##############
ALL_VERSIONS = ["4.2", "4.4", "5.0", "6.0", "7.0", "8.0", "rapid", "latest"]
CPYTHONS = ["3.10", "3.11", "3.12", "3.13", "3.14t", "3.14"]
PYPYS = ["pypy3.11"]
MIN_SUPPORT_VERSIONS = ["3.9", "pypy3.9", "pypy3.10"]
ALL_PYTHONS = CPYTHONS + PYPYS
MIN_MAX_PYTHON = [CPYTHONS[0], CPYTHONS[-1]]
BATCHTIME_WEEK = 10080
BATCHTIME_DAY = 1440
AUTH_SSLS = [("auth", "ssl"), ("noauth", "ssl"), ("noauth", "nossl")]
TOPOLOGIES = ["standalone", "replica_set", "sharded_cluster"]
C_EXTS = ["without_ext", "with_ext"]
SYNCS = ["sync", "async"]
DISPLAY_LOOKUP = dict(
ssl=dict(ssl="SSL", nossl="NoSSL"),
auth=dict(auth="Auth", noauth="NoAuth"),
topology=dict(
standalone="Standalone", replica_set="Replica Set", sharded_cluster="Sharded Cluster"
),
test_suites=dict(default="Sync", default_async="Async"),
sync={"sync": "Sync", "async": "Async"},
coverage={"1": "cov"},
no_ext={"1": "No C"},
test_min_deps={"1": "Min Deps"},
)
HOSTS = dict()
@dataclass
class Host:
name: str
run_on: str
display_name: str
variables: dict[str, str] | None
# Hosts with toolchains.
HOSTS["rhel8"] = Host("rhel8", "rhel87-small", "RHEL8", dict())
HOSTS["win64"] = Host("win64", "windows-64-vsMulti-small", "Win64", dict())
HOSTS["win-latest"] = Host("win-latest", "windows-2022-latest-small", "WinLatest", dict())
HOSTS["win32"] = Host("win32", "windows-64-vsMulti-small", "Win32", dict())
HOSTS["macos"] = Host("macos", "macos-14", "macOS", dict())
HOSTS["macos-arm64"] = Host("macos-arm64", "macos-14-arm64", "macOS Arm64", dict())
HOSTS["ubuntu22"] = Host("ubuntu22", "ubuntu2204-small", "Ubuntu-22", dict())
HOSTS["ubuntu24"] = Host("ubuntu24", "ubuntu2404-small", "Ubuntu-24", dict())
HOSTS["perf"] = Host("perf", "rhel90-dbx-perf-large", "", dict())
HOSTS["debian11"] = Host("debian11", "debian11-small", "Debian11", dict())
DEFAULT_HOST = HOSTS["rhel8"]
# Other hosts
OTHER_HOSTS = ["RHEL9-FIPS", "RHEL8-zseries", "RHEL8-POWER8", "RHEL8-arm64", "Amazon2023"]
for name, run_on in zip(
OTHER_HOSTS,
[
"rhel92-fips",
"rhel8-zseries-small",
"rhel8-power-small",
"rhel82-arm64-small",
"amazon2023-arm64-latest-large-m8g",
],
):
HOSTS[name] = Host(name, run_on, name, dict())
##############
# Helpers
##############
def create_variant_generic(
tasks: list[str | EvgTaskRef],
display_name: str,
*,
host: Host | str | None = None,
default_run_on="rhel87-small",
expansions: dict | None = None,
**kwargs: Any,
) -> BuildVariant:
"""Create a build variant for the given inputs."""
task_refs = []
if isinstance(host, str):
host = HOSTS[host]
for t in tasks:
if isinstance(t, EvgTaskRef):
task_refs.append(t)
else:
task_refs.append(EvgTaskRef(name=t))
expansions = expansions and expansions.copy() or dict()
if "run_on" in kwargs:
run_on = kwargs.pop("run_on")
elif host:
run_on = [host.run_on]
if host.variables:
expansions.update(host.variables)
else:
run_on = [default_run_on]
if isinstance(run_on, str):
run_on = [run_on]
name = display_name.replace(" ", "-").replace("*-", "").lower()
return BuildVariant(
name=name,
display_name=display_name,
tasks=task_refs,
expansions=expansions or None,
run_on=run_on,
**kwargs,
)
def create_variant(
tasks: list[str | EvgTaskRef],
display_name: str,
*,
version: str | None = None,
host: Host | str | None = None,
expansions: dict | None = None,
**kwargs: Any,
) -> BuildVariant:
expansions = expansions and expansions.copy() or dict()
if version:
expansions["VERSION"] = version
# 8.0+ Windows builds must run on win-latest
if (
"win64" in display_name.lower()
or "win32" in display_name.lower()
and version
and version >= "8.0"
):
kwargs["run_on"] = HOSTS["win-latest"].run_on
return create_variant_generic(
tasks, display_name, version=version, host=host, expansions=expansions, **kwargs
)
def get_versions_from(min_version: str) -> list[str]:
"""Get all server versions starting from a minimum version."""
min_version_float = float(min_version)
rapid_latest = ["rapid", "latest"]
versions = [v for v in ALL_VERSIONS if v not in rapid_latest]
return [v for v in versions if float(v) >= min_version_float] + rapid_latest
def get_versions_until(max_version: str) -> list[str]:
"""Get all server version up to a max version."""
max_version_float = float(max_version)
versions = [v for v in ALL_VERSIONS if v not in ["rapid", "latest"]]
versions = [v for v in versions if float(v) <= max_version_float]
if not len(versions):
raise ValueError(f"No server versions found less <= {max_version}")
return versions
def get_common_name(base: str, sep: str, **kwargs) -> str:
display_name = base
version = kwargs.pop("VERSION", None)
version = version or kwargs.pop("version", None)
if version:
if version not in ["rapid", "latest"]:
version = f"v{version}"
display_name = f"{display_name}{sep}{version}"
for key, value in kwargs.items():
name = value
if key.lower() in ["python", "toolchain_version"]:
if not value.startswith("pypy"):
name = f"Python{value}"
else:
name = f"PyPy{value.replace('pypy', '')}"
elif key.lower() in DISPLAY_LOOKUP and value in DISPLAY_LOOKUP[key.lower()]:
name = DISPLAY_LOOKUP[key.lower()][value]
else:
continue
display_name = f"{display_name}{sep}{name}"
return display_name
def get_variant_name(base: str, host: str | Host | None = None, **kwargs) -> str:
"""Get the display name of a variant."""
display_name = base
if isinstance(host, str):
host = HOSTS[host]
if host is not None:
display_name += f" {host.display_name}"
return get_common_name(display_name, " ", **kwargs)
def get_task_name(base: str, **kwargs):
return get_common_name(base, "-", **kwargs).replace(" ", "-").lower()
def zip_cycle(*iterables, empty_default=None):
"""Get all combinations of the inputs, cycling over the shorter list(s)."""
cycles = [cycle(i) for i in iterables]
for _ in zip_longest(*iterables):
yield tuple(next(i, empty_default) for i in cycles)
def handle_c_ext(c_ext, expansions) -> None:
"""Handle c extension option."""
if c_ext == C_EXTS[0]:
expansions["NO_EXT"] = "1"
def get_standard_auth_ssl(topology):
auth = "auth" if topology == "sharded_cluster" else "noauth"
ssl = "nossl" if topology == "standalone" else "ssl"
return auth, ssl
def get_assume_role(**kwargs):
kwargs.setdefault("command_type", EvgCommandType.SETUP)
kwargs.setdefault("role_arn", "${assume_role_arn}")
return ec2_assume_role(**kwargs)
def get_subprocess_exec(**kwargs):
kwargs.setdefault("binary", "bash")
kwargs.setdefault("working_dir", "src")
kwargs.setdefault("command_type", EvgCommandType.TEST)
return subprocess_exec(**kwargs)
def get_s3_put(**kwargs):
kwargs["aws_key"] = "${AWS_ACCESS_KEY_ID}"
kwargs["aws_secret"] = "${AWS_SECRET_ACCESS_KEY}" # noqa:S105
kwargs["aws_session_token"] = "${AWS_SESSION_TOKEN}" # noqa:S105
kwargs["bucket"] = "${bucket_name}"
kwargs.setdefault("optional", "true")
kwargs.setdefault("permissions", "public-read")
kwargs.setdefault("content_type", "${content_type|application/x-gzip}")
kwargs.setdefault("command_type", EvgCommandType.SETUP)
return s3_put(**kwargs)
def generate_yaml(tasks=None, variants=None):
"""Generate the yaml for a given set of tasks and variants."""
project = EvgProject(tasks=tasks, buildvariants=variants)
out = ShrubService.generate_yaml(project)
# Dedent by two spaces to match what we use in config.yml
lines = [line[2:] for line in out.splitlines()]
print("\n".join(lines))
##################
# Generate Config
##################
def write_variants_to_file(mod):
here = Path(__file__).absolute().parent
target = here.parent / "generated_configs" / "variants.yml"
if target.exists():
target.unlink()
with target.open("w") as fid:
fid.write("buildvariants:\n")
for name, func in sorted(getmembers(mod, isfunction)):
if not name.endswith("_variants"):
continue
if not name.startswith("create_"):
raise ValueError("Variant creators must start with create_")
title = name.replace("create_", "").replace("_variants", "").replace("_", " ").capitalize()
project = EvgProject(tasks=None, buildvariants=func())
out = ShrubService.generate_yaml(project).splitlines()
with target.open("a") as fid:
fid.write(f" # {title} tests\n")
for line in out[1:]:
fid.write(f"{line}\n")
fid.write("\n")
# Remove extra trailing newline:
data = target.read_text().splitlines()
with target.open("w") as fid:
for line in data[:-1]:
fid.write(f"{line}\n")
def write_tasks_to_file(mod):
here = Path(__file__).absolute().parent
target = here.parent / "generated_configs" / "tasks.yml"
if target.exists():
target.unlink()
with target.open("w") as fid:
fid.write("tasks:\n")
for name, func in sorted(getmembers(mod, isfunction)):
if name.startswith("_") or not name.endswith("_tasks"):
continue
if not name.startswith("create_"):
raise ValueError("Task creators must start with create_")
title = name.replace("create_", "").replace("_tasks", "").replace("_", " ").capitalize()
project = EvgProject(tasks=func(), buildvariants=None)
out = ShrubService.generate_yaml(project).splitlines()
with target.open("a") as fid:
fid.write(f" # {title} tests\n")
for line in out[1:]:
fid.write(f"{line}\n")
fid.write("\n")
# Remove extra trailing newline:
data = target.read_text().splitlines()
with target.open("w") as fid:
for line in data[:-1]:
fid.write(f"{line}\n")
def write_functions_to_file(mod):
here = Path(__file__).absolute().parent
target = here.parent / "generated_configs" / "functions.yml"
if target.exists():
target.unlink()
with target.open("w") as fid:
fid.write("functions:\n")
functions = dict()
for name, func in sorted(getmembers(mod, isfunction)):
if name.startswith("_") or not name.endswith("_func"):
continue
if not name.startswith("create_"):
raise ValueError("Function creators must start with create_")
title = name.replace("create_", "").replace("_func", "").replace("_", " ").capitalize()
func_name, cmds = func()
functions = dict()
functions[func_name] = cmds
project = EvgProject(functions=functions, tasks=None, buildvariants=None)
out = ShrubService.generate_yaml(project).splitlines()
with target.open("a") as fid:
fid.write(f" # {title}\n")
for line in out[1:]:
fid.write(f"{line}\n")
fid.write("\n")
# Remove extra trailing newline:
data = target.read_text().splitlines()
with target.open("w") as fid:
for line in data[:-1]:
fid.write(f"{line}\n")

View File

@ -1,36 +0,0 @@
#!/bin/bash
# Install the necessary dependencies.
set -eu
HERE=$(dirname ${BASH_SOURCE:-$0})
pushd "$(dirname "$(dirname $HERE)")" > /dev/null
# Source the env files to pick up common variables.
if [ -f $HERE/env.sh ]; then
. $HERE/env.sh
fi
# Set up the default bin directory.
if [ -z "${PYMONGO_BIN_DIR:-}" ]; then
PYMONGO_BIN_DIR="$HOME/.local/bin"
fi
# Ensure uv is installed.
if ! command -v uv &>/dev/null; then
_BIN_DIR=$PYMONGO_BIN_DIR
mkdir -p ${_BIN_DIR}
echo "Installing uv..."
curl -LsSf https://astral.sh/uv/install.sh | env UV_INSTALL_DIR="$_BIN_DIR" INSTALLER_NO_MODIFY_PATH=1 sh
if [ "Windows_NT" = "${OS:-}" ]; then
chmod +x "$(cygpath -u $_BIN_DIR)/uv.exe"
fi
export PATH="$PYMONGO_BIN_DIR:$PATH"
echo "Installing uv... done."
fi
# Ensure just is installed.
if ! command -v just &>/dev/null; then
uv tool install rust-just
fi
popd > /dev/null

View File

@ -1,144 +0,0 @@
from __future__ import annotations
import os
from utils import (
DRIVERS_TOOLS,
LOGGER,
TMP_DRIVER_FILE,
create_archive,
read_env,
run_command,
write_env,
)
DIRS = dict(
gcp=f"{DRIVERS_TOOLS}/.evergreen/csfle/gcpkms",
azure=f"{DRIVERS_TOOLS}/.evergreen/csfle/azurekms",
)
def _setup_azure_vm(base_env: dict[str, str]) -> None:
LOGGER.info("Setting up Azure VM...")
azure_dir = DIRS["azure"]
env = base_env.copy()
env["AZUREKMS_SRC"] = TMP_DRIVER_FILE
env["AZUREKMS_DST"] = "~/"
run_command(f"{azure_dir}/copy-file.sh", env=env)
env = base_env.copy()
env["AZUREKMS_CMD"] = "tar xf mongo-python-driver.tgz"
run_command(f"{azure_dir}/run-command.sh", env=env)
env["AZUREKMS_CMD"] = "sudo apt-get install -y python3-dev build-essential"
run_command(f"{azure_dir}/run-command.sh", env=env)
env["AZUREKMS_CMD"] = "bash .evergreen/just.sh setup-tests kms azure-remote"
run_command(f"{azure_dir}/run-command.sh", env=env)
LOGGER.info("Setting up Azure VM... done.")
def _setup_gcp_vm(base_env: dict[str, str]) -> None:
LOGGER.info("Setting up GCP VM...")
gcp_dir = DIRS["gcp"]
env = base_env.copy()
env["GCPKMS_SRC"] = TMP_DRIVER_FILE
env["GCPKMS_DST"] = f"{env['GCPKMS_INSTANCENAME']}:"
run_command(f"{gcp_dir}/copy-file.sh", env=env)
env = base_env.copy()
env["GCPKMS_CMD"] = "tar xf mongo-python-driver.tgz"
run_command(f"{gcp_dir}/run-command.sh", env=env)
env["GCPKMS_CMD"] = "sudo apt-get install -y python3-dev build-essential"
run_command(f"{gcp_dir}/run-command.sh", env=env)
env["GCPKMS_CMD"] = "bash ./.evergreen/just.sh setup-tests kms gcp-remote"
run_command(f"{gcp_dir}/run-command.sh", env=env)
LOGGER.info("Setting up GCP VM...")
def _load_kms_config(sub_test_target: str) -> dict[str, str]:
target_dir = DIRS[sub_test_target]
config = read_env(f"{target_dir}/secrets-export.sh")
base_env = os.environ.copy()
for key, value in config.items():
base_env[key] = str(value)
return base_env
def setup_kms(sub_test_name: str) -> None:
if "-" in sub_test_name:
sub_test_target, sub_test_type = sub_test_name.split("-")
else:
sub_test_target = sub_test_name
sub_test_type = ""
assert sub_test_target in ["azure", "gcp"], sub_test_target
assert sub_test_type in ["", "remote", "fail"], sub_test_type
success = sub_test_type != "fail"
kms_dir = DIRS[sub_test_target]
if sub_test_target == "azure":
write_env("TEST_FLE_AZURE_AUTO")
else:
write_env("TEST_FLE_GCP_AUTO")
write_env("SUCCESS", success)
# For remote tests, there is no further work required.
if sub_test_type == "remote":
return
if sub_test_target == "azure":
run_command("./setup-secrets.sh", cwd=kms_dir)
if success:
create_archive()
if sub_test_target == "azure":
os.environ["AZUREKMS_VMNAME_PREFIX"] = "PYTHON_DRIVER"
# Found using "az vm image list --output table"
os.environ[
"AZUREKMS_IMAGE"
] = "Canonical:0001-com-ubuntu-server-jammy:22_04-lts-gen2:latest"
else:
os.environ["GCPKMS_IMAGEFAMILY"] = "debian-12"
run_command("./setup.sh", cwd=kms_dir)
base_env = _load_kms_config(sub_test_target)
if sub_test_target == "azure":
_setup_azure_vm(base_env)
else:
_setup_gcp_vm(base_env)
if sub_test_target == "azure":
config = read_env(f"{kms_dir}/secrets-export.sh")
if success:
write_env("AZUREKMS_VMNAME", config["AZUREKMS_VMNAME"])
write_env("KEY_NAME", config["AZUREKMS_KEYNAME"])
write_env("KEY_VAULT_ENDPOINT", config["AZUREKMS_KEYVAULTENDPOINT"])
def test_kms_send_to_remote(sub_test_name: str) -> None:
env = _load_kms_config(sub_test_name)
if sub_test_name == "azure":
key_name = os.environ["KEY_NAME"]
key_vault_endpoint = os.environ["KEY_VAULT_ENDPOINT"]
env[
"AZUREKMS_CMD"
] = f'KEY_NAME="{key_name}" KEY_VAULT_ENDPOINT="{key_vault_endpoint}" bash ./.evergreen/just.sh run-tests'
else:
env["GCPKMS_CMD"] = "./.evergreen/just.sh run-tests"
cmd = f"{DIRS[sub_test_name]}/run-command.sh"
run_command(cmd, env=env)
def teardown_kms(sub_test_name: str) -> None:
run_command(f"{DIRS[sub_test_name]}/teardown.sh")
if __name__ == "__main__":
setup_kms()

View File

@ -1,93 +0,0 @@
from __future__ import annotations
import os
import sys
import time
import urllib.error
import urllib.request
from pathlib import Path
from shutil import which
from utils import LOGGER, ROOT, run_command, write_env
def make_request(url, timeout=10):
for _ in range(int(timeout)):
try:
urllib.request.urlopen(url) # noqa: S310
return
except urllib.error.HTTPError:
pass
time.sleep(1)
raise TimeoutError(f"Failed to access {url}")
def setup_mod_wsgi(sub_test_name: str) -> None:
env = os.environ.copy()
if sub_test_name == "embedded":
env["MOD_WSGI_CONF"] = "mod_wsgi_test_embedded.conf"
elif sub_test_name == "standalone":
env["MOD_WSGI_CONF"] = "mod_wsgi_test.conf"
else:
raise ValueError("mod_wsgi sub test must be either 'standalone' or 'embedded'")
write_env("MOD_WSGI_CONF", env["MOD_WSGI_CONF"])
apache = which("apache2")
if not apache and Path("/usr/lib/apache2/mpm-prefork/apache2").exists():
apache = "/usr/lib/apache2/mpm-prefork/apache2"
if apache:
apache_config = "apache24ubuntu161404.conf"
else:
apache = which("httpd")
if not apache:
raise ValueError("Could not find apache2 or httpd")
apache_config = "apache22amazon.conf"
python_version = ".".join(str(val) for val in sys.version_info[:2])
mod_wsgi_version = 4
so_file = f"/opt/python/mod_wsgi/python_version/{python_version}/mod_wsgi_version/{mod_wsgi_version}/mod_wsgi.so"
write_env("MOD_WSGI_SO", so_file)
env["MOD_WSGI_SO"] = so_file
env["PYTHONHOME"] = f"/opt/python/{python_version}"
env["PROJECT_DIRECTORY"] = project_directory = str(ROOT)
write_env("APACHE_BINARY", apache)
write_env("APACHE_CONFIG", apache_config)
uri1 = f"http://localhost:8080/interpreter1{project_directory}"
write_env("TEST_URI1", uri1)
uri2 = f"http://localhost:8080/interpreter2{project_directory}"
write_env("TEST_URI2", uri2)
run_command(f"{apache} -k start -f {ROOT}/test/mod_wsgi_test/{apache_config}", env=env)
# Wait for the endpoints to be available.
try:
make_request(uri1, 10)
make_request(uri2, 10)
except Exception as e:
LOGGER.error(Path("error_log").read_text())
raise e
def test_mod_wsgi() -> None:
sys.path.insert(0, ROOT)
from test.mod_wsgi_test.test_client import main, parse_args
uri1 = os.environ["TEST_URI1"]
uri2 = os.environ["TEST_URI2"]
args = f"-n 25000 -t 100 parallel {uri1} {uri2}"
try:
main(*parse_args(args.split()))
args = f"-n 25000 serial {uri1} {uri2}"
main(*parse_args(args.split()))
except Exception as e:
LOGGER.error(Path("error_log").read_text())
raise e
def teardown_mod_wsgi() -> None:
apache = os.environ["APACHE_BINARY"]
apache_config = os.environ["APACHE_CONFIG"]
run_command(f"{apache} -k stop -f {ROOT}/test/mod_wsgi_test/{apache_config}")
if __name__ == "__main__":
setup_mod_wsgi()

View File

@ -1,111 +0,0 @@
from __future__ import annotations
import os
from utils import (
DRIVERS_TOOLS,
TMP_DRIVER_FILE,
create_archive,
read_env,
run_command,
write_env,
)
K8S_NAMES = ["aks", "gke", "eks"]
K8S_REMOTE_NAMES = [f"{n}-remote" for n in K8S_NAMES]
def _get_target_dir(sub_test_name: str) -> str:
if sub_test_name == "default":
target_dir = "auth_oidc"
elif sub_test_name.startswith("azure"):
target_dir = "auth_oidc/azure"
elif sub_test_name.startswith("gcp"):
target_dir = "auth_oidc/gcp"
elif sub_test_name in K8S_NAMES + K8S_REMOTE_NAMES:
target_dir = "auth_oidc/k8s"
else:
raise ValueError(f"Invalid sub test name '{sub_test_name}'")
return f"{DRIVERS_TOOLS}/.evergreen/{target_dir}"
def setup_oidc(sub_test_name: str) -> dict[str, str] | None:
target_dir = _get_target_dir(sub_test_name)
env = os.environ.copy()
if sub_test_name == "eks" and "AWS_ACCESS_KEY_ID" in os.environ:
# Store AWS creds for kubectl access.
for key in ["AWS_ACCESS_KEY_ID", "AWS_SECRET_ACCESS_KEY", "AWS_SESSION_TOKEN"]:
if key in os.environ:
write_env(key, os.environ[key])
if sub_test_name == "azure":
env["AZUREOIDC_VMNAME_PREFIX"] = "PYTHON_DRIVER"
if "-remote" not in sub_test_name:
if sub_test_name == "azure":
# Found using "az vm image list --output table"
env["AZUREOIDC_IMAGE"] = "Canonical:0001-com-ubuntu-server-jammy:22_04-lts-gen2:latest"
else:
env["GCPKMS_IMAGEFAMILY"] = "debian-12"
run_command(f"bash {target_dir}/setup.sh", env=env)
if sub_test_name in K8S_NAMES:
run_command(f"bash {target_dir}/setup-pod.sh {sub_test_name}")
run_command(f"bash {target_dir}/run-self-test.sh")
return None
source_file = None
if sub_test_name == "default":
source_file = f"{target_dir}/secrets-export.sh"
elif sub_test_name in ["azure-remote", "gcp-remote"]:
source_file = "./secrets-export.sh"
if sub_test_name in K8S_REMOTE_NAMES:
return os.environ.copy()
if source_file is None:
return None
config = read_env(source_file)
write_env("MONGODB_URI_SINGLE", config["MONGODB_URI_SINGLE"])
write_env("MONGODB_URI", config["MONGODB_URI"])
write_env("DB_IP", config["MONGODB_URI"])
if sub_test_name == "default":
write_env("OIDC_TOKEN_FILE", config["OIDC_TOKEN_FILE"])
write_env("OIDC_TOKEN_DIR", config["OIDC_TOKEN_DIR"])
if "OIDC_DOMAIN" in config:
write_env("OIDC_DOMAIN", config["OIDC_DOMAIN"])
elif sub_test_name == "azure-remote":
write_env("AZUREOIDC_RESOURCE", config["AZUREOIDC_RESOURCE"])
elif sub_test_name == "gcp-remote":
write_env("GCPOIDC_AUDIENCE", config["GCPOIDC_AUDIENCE"])
return config
def test_oidc_send_to_remote(sub_test_name: str) -> None:
env = os.environ.copy()
target_dir = _get_target_dir(sub_test_name)
create_archive()
if sub_test_name in ["azure", "gcp"]:
upper_name = sub_test_name.upper()
env[f"{upper_name}OIDC_DRIVERS_TAR_FILE"] = TMP_DRIVER_FILE
env[
f"{upper_name}OIDC_TEST_CMD"
] = f"OIDC_ENV={sub_test_name} ./.evergreen/run-mongodb-oidc-test.sh"
elif sub_test_name in K8S_NAMES:
env["K8S_DRIVERS_TAR_FILE"] = TMP_DRIVER_FILE
env["K8S_TEST_CMD"] = "OIDC_ENV=k8s ./.evergreen/run-mongodb-oidc-test.sh"
run_command(f"bash {target_dir}/run-driver-test.sh", env=env)
def teardown_oidc(sub_test_name: str) -> None:
target_dir = _get_target_dir(sub_test_name)
# For k8s, make sure an error while tearing down the pod doesn't prevent
# the Altas server teardown.
error = None
if sub_test_name in K8S_NAMES:
try:
run_command(f"bash {target_dir}/teardown-pod.sh")
except Exception as e:
error = e
run_command(f"bash {target_dir}/teardown.sh")
if error:
raise error

View File

@ -1,15 +0,0 @@
#!/bin/bash
# We use the requester expansion to determine whether the data is from a mainline evergreen run or not
set -eu
# shellcheck disable=SC2154
if [ "${requester}" == "commit" ]; then
echo "is_mainline: true" >> expansion.yml
else
echo "is_mainline: false" >> expansion.yml
fi
# We parse the username out of the order_id as patches append that in and SPS does not need that information
# shellcheck disable=SC2154
echo "parsed_order_id: $(echo "${revision_order_id}" | awk -F'_' '{print $NF}')" >> expansion.yml

View File

@ -1,25 +0,0 @@
#!/bin/bash
# We use the requester expansion to determine whether the data is from a mainline evergreen run or not
set -eu
# Submit the performance data to the SPS endpoint
# shellcheck disable=SC2154
response=$(curl -s -w "\nHTTP_STATUS:%{http_code}" -X 'POST' \
"https://performance-monitoring-api.corp.mongodb.com/raw_perf_results/cedar_report?project=${project_id}&version=${version_id}&variant=${build_variant}&order=${parsed_order_id}&task_name=${task_name}&task_id=${task_id}&execution=${execution}&mainline=${is_mainline}" \
-H 'accept: application/json' \
-H 'Content-Type: application/json' \
-d @results.json)
http_status=$(echo "$response" | grep "HTTP_STATUS" | awk -F':' '{print $2}')
response_body=$(echo "$response" | sed '/HTTP_STATUS/d')
# We want to throw an error if the data was not successfully submitted
if [ "$http_status" -ne 200 ]; then
echo "Error: Received HTTP status $http_status"
echo "Response Body: $response_body"
exit 1
fi
echo "Response Body: $response_body"
echo "HTTP Status: $http_status"

View File

@ -1,150 +0,0 @@
from __future__ import annotations
import argparse
import os
import pathlib
import subprocess
from argparse import Namespace
from subprocess import CalledProcessError
JIRA_FILTER = "https://jira.mongodb.org/issues/?jql=labels%20%3D%20automated-sync%20AND%20status%20!%3D%20Closed"
def resync_specs(directory: pathlib.Path, errored: dict[str, str]) -> None:
"""Actually sync the specs"""
print("Beginning to sync specs")
for spec in os.scandir(directory):
if not spec.is_dir():
continue
if spec.name in ["asynchronous"]:
continue
try:
subprocess.run(
["bash", "./.evergreen/resync-specs.sh", spec.name], # noqa: S603, S607
capture_output=True,
text=True,
check=True,
)
except CalledProcessError as exc:
errored[spec.name] = exc.stderr
print("Done syncing specs")
def apply_patches(errored):
print("Beginning to apply patches")
subprocess.run(
["bash", "./.evergreen/remove-unimplemented-tests.sh"], # noqa: S603, S607
check=True,
)
try:
# Avoid shell=True by passing arguments as a list.
# Note: glob expansion doesn't work in shell=False, so we use a list of files.
patches = [str(p) for p in pathlib.Path("./.evergreen/spec-patch/").glob("*")]
if patches:
subprocess.run(
[ # noqa: S603, S607
"git",
"apply",
"-R",
"--allow-empty",
"--whitespace=fix",
*patches,
],
check=True,
stderr=subprocess.PIPE,
)
except CalledProcessError as exc:
errored["applying patches"] = exc.stderr
def check_new_spec_directories(directory: pathlib.Path) -> list[str]:
"""Check to see if there are any directories in the spec repo that don't exist in pymongo/test"""
spec_dir = pathlib.Path(os.environ["MDB_SPECS"]) / "source"
spec_set = {
entry.name.replace("-", "_")
for entry in os.scandir(spec_dir)
if entry.is_dir()
and (pathlib.Path(entry.path) / "tests").is_dir()
and len(list(os.scandir(pathlib.Path(entry.path) / "tests"))) > 1
}
test_set = {entry.name.replace("-", "_") for entry in os.scandir(directory) if entry.is_dir()}
known_mappings = {
"ocsp_support": "ocsp",
"client_side_operations_timeout": "csot",
"mongodb_handshake": "handshake",
"load_balancers": "load_balancer",
"connection_monitoring_and_pooling": "connection_monitoring",
"command_logging_and_monitoring": "command_logging",
"initial_dns_seedlist_discovery": "srv_seedlist",
"server_discovery_and_monitoring": "sdam_monitoring",
}
for k, v in known_mappings.items():
if k in spec_set:
spec_set.remove(k)
spec_set.add(v)
return list(spec_set - test_set)
def write_summary(errored: dict[str, str], new: list[str], filename: str | None) -> None:
"""Generate the PR description"""
pr_body = ""
# Avoid shell=True and complex pipes by using Python to process git output
process = subprocess.run(
["git", "diff", "--name-only"], # noqa: S603, S607
capture_output=True,
text=True,
check=True,
)
changed_files = process.stdout.strip().splitlines()
succeeded_set = set()
for f in changed_files:
parts = f.split("/")
if len(parts) > 1:
succeeded_set.add(parts[1])
succeeded = sorted(succeeded_set)
if len(succeeded) > 0:
pr_body += "The following specs were changed:\n -"
pr_body += "\n -".join(succeeded)
pr_body += "\n"
if len(errored) > 0:
pr_body += "\n\nThe following spec syncs encountered errors:"
for k, v in errored.items():
pr_body += f"\n -{k}\n```{v}\n```"
pr_body += "\n"
if len(new) > 0:
pr_body += "\n\nThe following directories are in the specification repository and not in our test directory:\n -"
pr_body += "\n -".join(new)
pr_body += "\n"
if pr_body != "":
pr_body = f"Jira tickets: {JIRA_FILTER}\n\n" + pr_body
if filename is None:
print(f"\n{pr_body}")
else:
with open(filename, "w") as f:
# replacements made for proper json
f.write(pr_body.replace("\n", "\\n").replace("\t", "\\t"))
def main(args: Namespace):
directory = pathlib.Path("./test")
errored: dict[str, str] = {}
resync_specs(directory, errored)
apply_patches(errored)
new = check_new_spec_directories(directory)
write_summary(errored, new, args.filename)
if __name__ == "__main__":
parser = argparse.ArgumentParser(
description="Python Script to resync all specs and generate summary for PR."
)
parser.add_argument(
"--filename",
help="Name of file for the summary to be written into.",
default=None,
)
args = parser.parse_args()
main(args)

View File

@ -1,43 +0,0 @@
#!/usr/bin/env bash
# Run spec syncing script and create PR
set -eu
# SETUP
SRC_URL="https://github.com/mongodb/specifications.git"
# needs to be set for resync-specs.sh
SPEC_SRC="$(realpath "../specifications")"
SCRIPT="$(realpath "./.evergreen/resync-specs.sh")"
# Clone the spec repo if the directory does not exist
if [[ ! -d $SPEC_SRC ]]; then
git clone $SRC_URL $SPEC_SRC
if [[ $? -ne 0 ]]; then
echo "Error: Failed to clone repository."
exit 1
fi
fi
# Set environment variable to the cloned spec repo for resync-specs.sh
export MDB_SPECS="$SPEC_SRC"
# Check that resync-specs.sh exists and is executable
if [[ ! -x $SCRIPT ]]; then
echo "Error: $SCRIPT not found or is not executable."
exit 1
fi
PR_DESC="spec_sync.txt"
# run python script that actually does all the resyncing
if ! [ -n "${CI:-}" ]
then
# we're running locally
python3 ./.evergreen/scripts/resync-all-specs.py
else
/opt/devtools/bin/python3.11 ./.evergreen/scripts/resync-all-specs.py --filename "$PR_DESC"
if [[ -f $PR_DESC ]]; then
# changes were made -> call scrypt to create PR for us
.evergreen/scripts/create-spec-pr.sh "$PR_DESC"
rm "$PR_DESC"
fi
fi

View File

@ -1,26 +0,0 @@
#!/bin/bash
# Get the debug data for an evergreen task.
set -eu
. ${DRIVERS_TOOLS}/.evergreen/get-distro.sh || true
get_distro || true
echo $DISTRO
echo $MARCH
echo $OS
set -x
uname -a || true
ls /etc/*release* || true
cc --version || true
gcc --version || true
clang --version || true
gcov --version || true
lcov --version || true
llvm-cov --version || true
echo $PATH
ls -la /usr/local/Cellar/llvm/*/bin/ || true
ls -la /usr/local/Cellar/ || true
scan-build --version || true
genhtml --version || true
valgrind --version || true
set +x

View File

@ -1,13 +0,0 @@
#!/bin/bash
set -eu
HERE=$(dirname ${BASH_SOURCE:-$0})
# Try to source the env file.
if [ -f $HERE/env.sh ]; then
echo "Sourcing env file"
source $HERE/env.sh
fi
uv run $HERE/run_server.py "$@"

View File

@ -1,59 +0,0 @@
from __future__ import annotations
import os
from typing import Any
from utils import DRIVERS_TOOLS, ROOT, get_test_options, run_command
def set_env(name: str, value: Any = "1") -> None:
os.environ[name] = str(value)
def start_server():
opts, extra_opts = get_test_options(
"Run a MongoDB server. All given flags will be passed to run-mongodb.sh in DRIVERS_TOOLS.",
require_sub_test_name=False,
allow_extra_opts=True,
)
test_name = opts.test_name
# drivers-evergreen-tools expects the version variable to be named MONGODB_VERSION.
if "VERSION" in os.environ:
os.environ["MONGODB_VERSION"] = os.environ["VERSION"]
if test_name == "auth_aws":
set_env("AUTH_AWS")
elif test_name == "load_balancer":
set_env("LOAD_BALANCER")
elif test_name == "search_index":
os.environ["TOPOLOGY"] = "replica_set"
os.environ["MONGODB_VERSION"] = "7.0"
if not os.environ.get("TEST_CRYPT_SHARED"):
set_env("SKIP_CRYPT_SHARED")
if opts.ssl:
extra_opts.append("--ssl")
if test_name != "ocsp":
certs = ROOT / "test/certificates"
set_env("TLS_CERT_KEY_FILE", certs / "client.pem")
set_env("TLS_PEM_KEY_FILE", certs / "server.pem")
set_env("TLS_CA_FILE", certs / "ca.pem")
if opts.auth:
extra_opts.append("--auth")
if opts.verbose:
extra_opts.append("-v")
elif opts.quiet:
extra_opts.append("-q")
cmd = ["bash", f"{DRIVERS_TOOLS}/.evergreen/run-mongodb.sh", "start", *extra_opts]
run_command(cmd, cwd=DRIVERS_TOOLS)
if __name__ == "__main__":
start_server()

View File

@ -1,228 +0,0 @@
from __future__ import annotations
import json
import logging
import os
import platform
import shlex
import shutil
import subprocess
import sys
from datetime import datetime
from pathlib import Path
from shutil import which
try:
import importlib_metadata
except ImportError:
from importlib import metadata as importlib_metadata
import pytest
from utils import DRIVERS_TOOLS, LOGGER, ROOT, run_command
AUTH = os.environ.get("AUTH", "noauth")
SSL = os.environ.get("SSL", "nossl")
UV_ARGS = os.environ.get("UV_ARGS", "")
TEST_PERF = os.environ.get("TEST_PERF")
GREEN_FRAMEWORK = os.environ.get("GREEN_FRAMEWORK")
TEST_ARGS = os.environ.get("TEST_ARGS", "").split()
TEST_NAME = os.environ.get("TEST_NAME")
SUB_TEST_NAME = os.environ.get("SUB_TEST_NAME")
def list_packages():
packages = set()
for distribution in importlib_metadata.distributions():
if distribution.name:
packages.add(distribution.name)
print("Package Version URL")
print("------------------- ----------- ----------------------------------------------------")
for name in sorted(packages):
distribution = importlib_metadata.distribution(name)
url = ""
if distribution.origin is not None:
url = distribution.origin.url
print(f"{name:20s}{distribution.version:12s}{url}")
print("------------------- ----------- ----------------------------------------------------\n")
def handle_perf(start_time: datetime):
end_time = datetime.now()
elapsed_secs = (end_time - start_time).total_seconds()
with open("results.json") as fid:
results = json.load(fid)
LOGGER.info("results.json:\n%s", json.dumps(results, indent=2))
results = dict(
status="PASS",
exit_code=0,
test_file="BenchMarkTests",
start=int(start_time.timestamp()),
end=int(end_time.timestamp()),
elapsed=elapsed_secs,
)
report = dict(failures=0, results=[results])
LOGGER.info("report.json\n%s", json.dumps(report, indent=2))
with open("report.json", "w", newline="\n") as fid:
json.dump(report, fid)
def handle_green_framework() -> None:
if GREEN_FRAMEWORK == "gevent":
from gevent import monkey
monkey.patch_all()
# Never run async tests with a framework.
if len(TEST_ARGS) <= 1:
TEST_ARGS.extend(["-m", "not default_async and default"])
else:
for i in range(len(TEST_ARGS) - 1):
if "-m" in TEST_ARGS[i]:
TEST_ARGS[i + 1] = f"not default_async and {TEST_ARGS[i + 1]}"
LOGGER.info(f"Running tests with {GREEN_FRAMEWORK}...")
def handle_c_ext() -> None:
if platform.python_implementation() != "CPython":
return
sys.path.insert(0, str(ROOT / "tools"))
from fail_if_no_c import main as fail_if_no_c
fail_if_no_c()
def handle_pymongocrypt() -> None:
import pymongocrypt
LOGGER.info(f"pymongocrypt version: {pymongocrypt.__version__})")
LOGGER.info(f"libmongocrypt version: {pymongocrypt.libmongocrypt_version()})")
def handle_aws_lambda() -> None:
env = os.environ.copy()
target_dir = ROOT / "test/lambda"
env["TEST_LAMBDA_DIRECTORY"] = str(target_dir)
env.setdefault("AWS_REGION", "us-east-1")
dirs = ["pymongo", "gridfs", "bson"]
# Remove the original .so files.
for dname in dirs:
so_paths = [f"{f.parent.name}/{f.name}" for f in (ROOT / dname).glob("*.so")]
for so_path in list(so_paths):
Path(so_path).unlink()
# Build the c extensions.
docker = which("docker") or which("podman")
if not docker:
raise ValueError("Could not find docker!")
image = "quay.io/pypa/manylinux2014_x86_64:latest"
run_command(
f'{docker} run --rm -v "{ROOT}:/src" --platform linux/amd64 {image} /src/test/lambda/build_internal.sh'
)
for dname in dirs:
target = ROOT / "test/lambda/mongodb" / dname
shutil.rmtree(target, ignore_errors=True)
shutil.copytree(ROOT / dname, target)
# Remove the new so files from the ROOT directory.
for dname in dirs:
so_paths = [f"{f.parent.name}/{f.name}" for f in (ROOT / dname).glob("*.so")]
for so_path in list(so_paths):
Path(so_path).unlink()
script_name = "run-deployed-lambda-aws-tests.sh"
run_command(f"bash {DRIVERS_TOOLS}/.evergreen/aws_lambda/{script_name}", env=env)
def run() -> None:
# Add diagnostic for python version.
print("Running with python", sys.version)
# List the installed packages.
list_packages()
# Handle green framework first so they can patch modules.
if GREEN_FRAMEWORK:
handle_green_framework()
# Ensure C extensions if applicable.
if not os.environ.get("NO_EXT"):
handle_c_ext()
if os.environ.get("PYMONGOCRYPT_LIB"):
handle_pymongocrypt()
LOGGER.info(f"Test setup:\n{AUTH=}\n{SSL=}\n{UV_ARGS=}\n{TEST_ARGS=}")
# Record the start time for a perf test.
if TEST_PERF:
start_time = datetime.now()
# Run mod_wsgi tests using the helper.
if TEST_NAME == "mod_wsgi":
from mod_wsgi_tester import test_mod_wsgi
test_mod_wsgi()
return
# Send kms tests to run remotely.
if TEST_NAME == "kms" and SUB_TEST_NAME in ["azure", "gcp"]:
from kms_tester import test_kms_send_to_remote
test_kms_send_to_remote(SUB_TEST_NAME)
return
# Handle doctests.
if TEST_NAME == "doctest":
from sphinx.cmd.build import main
result = main("-E -b doctest doc ./doc/_build/doctest".split())
sys.exit(result)
# Send ecs tests to run remotely.
if TEST_NAME == "auth_aws" and SUB_TEST_NAME == "ecs":
run_command(f"{DRIVERS_TOOLS}/.evergreen/auth_aws/aws_setup.sh ecs")
return
# Send OIDC tests to run remotely.
if (
TEST_NAME == "auth_oidc"
and SUB_TEST_NAME != "default"
and not SUB_TEST_NAME.endswith("-remote")
):
from oidc_tester import test_oidc_send_to_remote
test_oidc_send_to_remote(SUB_TEST_NAME)
return
# Run deployed aws lambda tests.
if TEST_NAME == "aws_lambda":
handle_aws_lambda()
return
if os.environ.get("DEBUG_LOG"):
TEST_ARGS.extend(f"-o log_cli_level={logging.DEBUG}".split())
if os.environ.get("COVERAGE"):
binary = sys.executable.replace(os.sep, "/")
cmd = f"{binary} -m coverage run -m pytest {' '.join(TEST_ARGS)} {' '.join(sys.argv[1:])}"
result = subprocess.run(shlex.split(cmd), check=False) # noqa: S603
cmd = f"{binary} -m coverage report"
subprocess.run(shlex.split(cmd), check=False) # noqa: S603
if result.returncode != 0:
print(result.stderr)
sys.exit(result.returncode)
# Run local tests.
ret = pytest.main(TEST_ARGS + sys.argv[1:])
if ret != 0:
sys.exit(ret)
# Handle perf test post actions.
if TEST_PERF:
handle_perf(start_time)
if __name__ == "__main__":
run()

View File

@ -1,58 +0,0 @@
#!/bin/bash
# Set up development environment.
set -eu
HERE=$(dirname ${BASH_SOURCE:-$0})
HERE="$( cd -- "$HERE" > /dev/null 2>&1 && pwd )"
ROOT=$(dirname "$(dirname $HERE)")
# Source the env files to pick up common variables.
if [ -f $HERE/env.sh ]; then
. $HERE/env.sh
fi
# Get variables defined in test-env.sh.
if [ -f $HERE/test-env.sh ]; then
. $HERE/test-env.sh
fi
# Ensure dependencies are installed.
bash $HERE/install-dependencies.sh
# Handle the value for UV_PYTHON.
. $HERE/setup-uv-python.sh
# Only run the next part if not running on CI.
if [ -z "${CI:-}" ]; then
# Add the default install path to the path if needed.
if [ -z "${PYMONGO_BIN_DIR:-}" ]; then
export PATH="$PATH:$HOME/.local/bin"
fi
# Set up venv, making sure c extensions build unless disabled.
if [ -z "${NO_EXT:-}" ]; then
export PYMONGO_C_EXT_MUST_BUILD=1
fi
(
cd $ROOT && uv sync
)
# Set up build utilities on Windows spawn hosts.
if [ -f $HOME/.visualStudioEnv.sh ]; then
set +u
SSH_TTY=1 source $HOME/.visualStudioEnv.sh
set -u
fi
# Only set up pre-commit if we are in a git checkout.
if [ -f $HERE/.git ]; then
if ! command -v pre-commit &>/dev/null; then
uv tool install pre-commit
fi
if [ ! -f .git/hooks/pre-commit ]; then
uvx pre-commit install
fi
fi
fi

View File

@ -1,55 +0,0 @@
#!/bin/bash
# Set up the system on an evergreen host.
set -eu
HERE=$(dirname ${BASH_SOURCE:-$0})
pushd "$(dirname "$(dirname $HERE)")"
echo "Setting up system..."
bash .evergreen/scripts/configure-env.sh
source .evergreen/scripts/env.sh
bash $DRIVERS_TOOLS/.evergreen/setup.sh
popd
# Run spawn host-specific tasks.
if [ -z "${CI:-}" ]; then
bash $HERE/setup-dev-env.sh
fi
# Enable core dumps if enabled on the machine
# Copied from https://github.com/mongodb/mongo/blob/master/etc/evergreen.yml
if [ -f /proc/self/coredump_filter ]; then
# Set the shell process (and its children processes) to dump ELF headers (bit 4),
# anonymous shared mappings (bit 1), and anonymous private mappings (bit 0).
echo 0x13 >/proc/self/coredump_filter
if [ -f /sbin/sysctl ]; then
# Check that the core pattern is set explicitly on our distro image instead
# of being the OS's default value. This ensures that coredump names are consistent
# across distros and can be picked up by Evergreen.
core_pattern=$(/sbin/sysctl -n "kernel.core_pattern")
if [ "$core_pattern" = "dump_%e.%p.core" ]; then
echo "Enabling coredumps"
ulimit -c unlimited
fi
fi
fi
if [ "$(uname -s)" = "Darwin" ]; then
core_pattern_mac=$(/usr/sbin/sysctl -n "kern.corefile")
if [ "$core_pattern_mac" = "dump_%N.%P.core" ]; then
echo "Enabling coredumps"
ulimit -c unlimited
fi
fi
if [ -w /etc/hosts ]; then
SUDO=""
else
SUDO="sudo"
fi
# Add 'server' and 'hostname_not_in_cert' as a hostnames
echo "127.0.0.1 server" | $SUDO tee -a /etc/hosts
echo "127.0.0.1 hostname_not_in_cert" | $SUDO tee -a /etc/hosts
echo "Setting up system... done."

View File

@ -1,26 +0,0 @@
#!/bin/bash
# Set up the test environment, including secrets and services.
set -eu
# Supported/used environment variables:
# AUTH Set to enable authentication. Defaults to "noauth"
# SSL Set to enable SSL. Defaults to "nossl"
# GREEN_FRAMEWORK The green framework to test with, if any.
# COVERAGE If non-empty, run the test suite with coverage.
# COMPRESSORS If non-empty, install appropriate compressor.
# LIBMONGOCRYPT_URL The URL to download libmongocrypt.
# TEST_CRYPT_SHARED If non-empty, install crypt_shared lib.
# MONGODB_API_VERSION The mongodb api version to use in tests.
# MONGODB_URI If non-empty, use as the MONGODB_URI in tests.
# USE_ACTIVE_VENV If non-empty, use the active virtual environment.
SCRIPT_DIR=$(dirname ${BASH_SOURCE:-$0})
# Try to source the env file.
if [ -f $SCRIPT_DIR/env.sh ]; then
source $SCRIPT_DIR/env.sh
fi
echo "Setting up tests with args \"$*\"..."
uv run ${USE_ACTIVE_VENV:+--active} "$SCRIPT_DIR/setup_tests.py" "$@"
echo "Setting up tests with args \"$*\"... done."

View File

@ -1,53 +0,0 @@
#!/bin/bash
# Set up the UV_PYTHON variable.
set -eu
HERE=$(dirname ${BASH_SOURCE:-$0})
HERE="$( cd -- "$HERE" > /dev/null 2>&1 && pwd )"
# Use min supported version by default.
_python="3.10"
# Source the env files to pick up common variables.
if [ -f $HERE/env.sh ]; then
. $HERE/env.sh
fi
# Get variables defined in test-env.sh.
if [ -f $HERE/test-env.sh ]; then
. $HERE/test-env.sh
fi
if [ -z "${UV_PYTHON:-}" ]; then
set -x
# Translate a TOOLCHAIN_VERSION to UV_PYTHON.
if [ -n "${TOOLCHAIN_VERSION:-}" ]; then
_python=$TOOLCHAIN_VERSION
if [ "$(uname -s)" = "Darwin" ]; then
if [[ "$_python" == *"t"* ]]; then
binary_name="python3t"
framework_dir="PythonT"
else
binary_name="python3"
framework_dir="Python"
fi
_python=$(echo "$_python" | sed 's/t//g')
_python="/Library/Frameworks/$framework_dir.Framework/Versions/$_python/bin/$binary_name"
elif [ "Windows_NT" = "${OS:-}" ]; then
_python=$(echo $_python | cut -d. -f1,2 | sed 's/\.//g; s/t//g')
if [[ "$TOOLCHAIN_VERSION" == *"t"* ]]; then
_exe="python${TOOLCHAIN_VERSION}.exe"
else
_exe="python.exe"
fi
if [ -n "${IS_WIN32:-}" ]; then
_python="C:/python/32/Python${_python}/${_exe}"
else
_python="C:/python/Python${_python}/${_exe}"
fi
elif [ -d "/opt/python/$_python/bin" ]; then
_python="/opt/python/$_python/bin/python3"
fi
fi
export UV_PYTHON="$_python"
fi

View File

@ -1,491 +0,0 @@
from __future__ import annotations
import base64
import os
import platform
import shutil
import stat
from pathlib import Path
from urllib import request
from utils import (
DRIVERS_TOOLS,
ENV_FILE,
HERE,
LOGGER,
PLATFORM,
ROOT,
TEST_SUITE_MAP,
Distro,
get_test_options,
read_env,
run_command,
write_env,
)
# Passthrough environment variables.
PASS_THROUGH_ENV = [
"GREEN_FRAMEWORK",
"NO_EXT",
"MONGODB_API_VERSION",
"DEBUG_LOG",
"UV_PYTHON",
"REQUIRE_FIPS",
"IS_WIN32",
]
# Map the test name to test extra.
EXTRAS_MAP = {
"auth_aws": "aws",
"auth_oidc": "aws",
"encryption": "encryption",
"enterprise_auth": "gssapi",
"kms": "encryption",
"ocsp": "ocsp",
"pyopenssl": "ocsp",
}
# Map the test name to test group.
GROUP_MAP = dict(mockupdb="mockupdb", perf="perf")
# The python version used for perf tests.
PERF_PYTHON_VERSION = "3.10.11"
def is_set(var: str) -> bool:
value = os.environ.get(var, "")
return len(value.strip()) > 0
def get_distro() -> Distro:
name = ""
version_id = ""
arch = platform.machine()
with open("/etc/os-release") as fid:
for line in fid.readlines():
line = line.replace('"', "") # noqa: PLW2901
if line.startswith("NAME="):
_, _, name = line.strip().partition("=")
if line.startswith("VERSION_ID="):
_, _, version_id = line.strip().partition("=")
return Distro(name=name, version_id=version_id, arch=arch)
def setup_libmongocrypt():
target = ""
if PLATFORM == "windows":
# PYTHON-2808 Ensure this machine has the CA cert for google KMS.
if is_set("TEST_FLE_GCP_AUTO"):
run_command('powershell.exe "Invoke-WebRequest -URI https://oauth2.googleapis.com/"')
target = "windows-test"
elif PLATFORM == "darwin":
target = "macos"
else:
distro = get_distro()
if distro.name.startswith("Debian"):
target = f"debian{distro.version_id}"
elif distro.name.startswith("Ubuntu"):
if distro.version_id == "20.04":
target = "debian11"
elif distro.version_id == "22.04":
target = "debian12"
elif distro.version_id == "24.04":
target = "debian13"
elif distro.name.startswith("Red Hat"):
if distro.version_id.startswith("7"):
target = "rhel-70-64-bit"
elif distro.version_id.startswith("8"):
if distro.arch == "aarch64":
target = "rhel-82-arm64"
else:
target = "rhel-80-64-bit"
if not is_set("LIBMONGOCRYPT_URL"):
if not target:
raise ValueError("Cannot find libmongocrypt target for current platform!")
url = f"https://s3.amazonaws.com/mciuploads/libmongocrypt/{target}/master/latest/libmongocrypt.tar.gz"
else:
url = os.environ["LIBMONGOCRYPT_URL"]
shutil.rmtree(HERE / "libmongocrypt", ignore_errors=True)
LOGGER.info(f"Fetching {url}...")
with request.urlopen(request.Request(url), timeout=15.0) as response: # noqa: S310
if response.status == 200:
with Path("libmongocrypt.tar.gz").open("wb") as f:
f.write(response.read())
Path("libmongocrypt").mkdir()
run_command("tar -xzf libmongocrypt.tar.gz -C libmongocrypt")
LOGGER.info(f"Fetching {url}... done.")
run_command("ls -la libmongocrypt")
run_command("ls -la libmongocrypt/nocrypto")
if PLATFORM == "windows":
# libmongocrypt's windows dll is not marked executable.
run_command("chmod +x libmongocrypt/nocrypto/bin/mongocrypt.dll")
def load_config_from_file(path: str | Path) -> dict[str, str]:
config = read_env(path)
for key, value in config.items():
write_env(key, value)
return config
def get_secrets(name: str) -> dict[str, str]:
secrets_dir = Path(f"{DRIVERS_TOOLS}/.evergreen/secrets_handling")
run_command(f"bash {secrets_dir.as_posix()}/setup-secrets.sh {name}", cwd=secrets_dir)
return load_config_from_file(secrets_dir / "secrets-export.sh")
def handle_test_env() -> None:
opts, _ = get_test_options("Set up the test environment and services.")
test_name = opts.test_name
sub_test_name = opts.sub_test_name
AUTH = "auth" if opts.auth else "noauth"
SSL = "ssl" if opts.ssl else "nossl"
TEST_ARGS = ""
# Start compiling the args we'll pass to uv.
UV_ARGS = ["--extra test --no-group dev"]
# If USE_ACTIVE_VENV is set, add --active to UV_ARGS so run-tests.sh uses the active venv.
if is_set("USE_ACTIVE_VENV"):
UV_ARGS.append("--active")
test_title = test_name
if sub_test_name:
test_title += f" {sub_test_name}"
# Create the test env file with the initial set of values.
with ENV_FILE.open("w", newline="\n") as fid:
fid.write("#!/usr/bin/env bash\n")
fid.write("set +x\n")
ENV_FILE.chmod(ENV_FILE.stat().st_mode | stat.S_IEXEC)
write_env("PIP_QUIET") # Quiet by default.
write_env("PIP_PREFER_BINARY") # Prefer binary dists by default.
# Set an environment variable for the test name and sub test name.
write_env(f"TEST_{test_name.upper()}")
write_env("TEST_NAME", test_name)
write_env("SUB_TEST_NAME", sub_test_name)
# Handle pass through env vars.
for var in PASS_THROUGH_ENV:
if is_set(var) or getattr(opts, var.lower(), ""):
write_env(var, os.environ.get(var, getattr(opts, var.lower(), "")))
if extra := EXTRAS_MAP.get(test_name, ""):
UV_ARGS.append(f"--extra {extra}")
if group := GROUP_MAP.get(test_name, ""):
UV_ARGS.append(f"--group {group}")
if opts.test_min_deps:
UV_ARGS.append("--resolution=lowest-direct")
if test_name == "auth_oidc":
from oidc_tester import setup_oidc
config = setup_oidc(sub_test_name)
if not config:
AUTH = "noauth"
if test_name in ["aws_lambda", "search_index"]:
env = os.environ.copy()
env["MONGODB_VERSION"] = "7.0"
env["LAMBDA_STACK_NAME"] = "dbx-python-lambda"
write_env("LAMBDA_STACK_NAME", env["LAMBDA_STACK_NAME"])
run_command(
f"bash {DRIVERS_TOOLS}/.evergreen/atlas/setup-atlas-cluster.sh",
env=env,
cwd=DRIVERS_TOOLS,
)
if test_name == "search_index":
AUTH = "auth"
if test_name == "ocsp":
SSL = "ssl"
write_env("AUTH", AUTH)
write_env("SSL", SSL)
LOGGER.info(f"Setting up '{test_title}' with {AUTH=} and {SSL=}...")
if test_name == "aws_lambda":
UV_ARGS.append("--group pip")
# Store AWS creds if they were given.
if "AWS_ACCESS_KEY_ID" in os.environ:
for key in ["AWS_ACCESS_KEY_ID", "AWS_SECRET_ACCESS_KEY", "AWS_SESSION_TOKEN"]:
if key in os.environ:
write_env(key, os.environ[key])
if AUTH != "noauth":
if test_name == "auth_oidc":
DB_USER = config["OIDC_ADMIN_USER"]
DB_PASSWORD = config["OIDC_ADMIN_PWD"]
elif test_name == "search_index":
config = read_env(f"{DRIVERS_TOOLS}/.evergreen/atlas/secrets-export.sh")
DB_USER = config["DRIVERS_ATLAS_LAMBDA_USER"]
DB_PASSWORD = config["DRIVERS_ATLAS_LAMBDA_PASSWORD"]
write_env("MONGODB_URI", config["MONGODB_URI"])
else:
DB_USER = "bob"
DB_PASSWORD = "pwd123" # noqa: S105
write_env("DB_USER", DB_USER)
write_env("DB_PASSWORD", DB_PASSWORD)
LOGGER.info("Added auth, DB_USER: %s", DB_USER)
if is_set("MONGODB_URI"):
write_env("PYMONGO_MUST_CONNECT", "true")
if opts.disable_test_commands:
write_env("PYMONGO_DISABLE_TEST_COMMANDS", "1")
if test_name == "enterprise_auth":
config = get_secrets("drivers/enterprise_auth")
if PLATFORM == "windows":
LOGGER.info("Setting GSSAPI_PASS")
write_env("GSSAPI_PASS", config["SASL_PASS"])
write_env("GSSAPI_CANONICALIZE", "true")
else:
# BUILD-3830
krb_conf = ROOT / ".evergreen/krb5.conf.empty"
krb_conf.touch()
write_env("KRB5_CONFIG", krb_conf)
LOGGER.info("Writing keytab")
keytab = base64.b64decode(config["KEYTAB_BASE64"])
keytab_file = ROOT / ".evergreen/drivers.keytab"
with keytab_file.open("wb") as fid:
fid.write(keytab)
principal = config["PRINCIPAL"]
LOGGER.info("Running kinit")
os.environ["KRB5_CONFIG"] = str(krb_conf)
cmd = f"kinit -k -t {keytab_file} -p {principal}"
run_command(cmd)
LOGGER.info("Setting GSSAPI variables")
write_env("GSSAPI_HOST", config["SASL_HOST"])
write_env("GSSAPI_PORT", config["SASL_PORT"])
write_env("GSSAPI_PRINCIPAL", config["PRINCIPAL"])
if test_name == "doctest":
UV_ARGS.append("--extra docs")
if test_name == "load_balancer":
SINGLE_MONGOS_LB_URI = os.environ.get(
"SINGLE_MONGOS_LB_URI", "mongodb://127.0.0.1:8000/?loadBalanced=true"
)
MULTI_MONGOS_LB_URI = os.environ.get(
"MULTI_MONGOS_LB_URI", "mongodb://127.0.0.1:8001/?loadBalanced=true"
)
if SSL != "nossl":
SINGLE_MONGOS_LB_URI += "&tls=true"
MULTI_MONGOS_LB_URI += "&tls=true"
write_env("SINGLE_MONGOS_LB_URI", SINGLE_MONGOS_LB_URI)
write_env("MULTI_MONGOS_LB_URI", MULTI_MONGOS_LB_URI)
if not DRIVERS_TOOLS:
raise RuntimeError("Missing DRIVERS_TOOLS")
cmd = f'bash "{DRIVERS_TOOLS}/.evergreen/run-load-balancer.sh" start'
run_command(cmd)
if test_name == "mod_wsgi":
from mod_wsgi_tester import setup_mod_wsgi
setup_mod_wsgi(sub_test_name)
if test_name == "ocsp":
if sub_test_name:
os.environ["OCSP_SERVER_TYPE"] = sub_test_name
for name in ["OCSP_SERVER_TYPE", "ORCHESTRATION_FILE"]:
if name not in os.environ:
raise ValueError(f"Please set {name}")
server_type = os.environ["OCSP_SERVER_TYPE"]
orch_file = os.environ["ORCHESTRATION_FILE"]
ocsp_algo = orch_file.split("-")[0]
if server_type == "no-responder":
tls_should_succeed = "false" if "mustStaple-disableStapling" in orch_file else "true"
else:
tls_should_succeed = "true" if "valid" in server_type else "false"
write_env("OCSP_TLS_SHOULD_SUCCEED", tls_should_succeed)
write_env("CA_FILE", f"{DRIVERS_TOOLS}/.evergreen/ocsp/{ocsp_algo}/ca.pem")
if server_type != "no-responder":
env = os.environ.copy()
env["SERVER_TYPE"] = server_type
env["OCSP_ALGORITHM"] = ocsp_algo
run_command(f"bash {DRIVERS_TOOLS}/.evergreen/ocsp/setup.sh", env=env)
# The mock OCSP responder MUST BE started before the mongod as the mongod expects that
# a responder will be available upon startup.
version = os.environ.get("VERSION", "latest")
cmd = [
"bash",
f"{DRIVERS_TOOLS}/.evergreen/run-mongodb.sh",
"start",
"--ssl",
"--version",
version,
]
if opts.verbose:
cmd.append("-v")
elif opts.quiet:
cmd.append("-q")
run_command(cmd, cwd=DRIVERS_TOOLS)
if SSL != "nossl":
if not DRIVERS_TOOLS:
raise RuntimeError("Missing DRIVERS_TOOLS")
write_env("CLIENT_PEM", f"{DRIVERS_TOOLS}/.evergreen/x509gen/client.pem")
write_env("CA_PEM", f"{DRIVERS_TOOLS}/.evergreen/x509gen/ca.pem")
compressors = os.environ.get("COMPRESSORS") or opts.compressor
if compressors == "snappy":
UV_ARGS.append("--extra snappy")
elif compressors == "zstd":
UV_ARGS.append("--extra zstd")
if test_name in ["encryption", "kms"]:
# Check for libmongocrypt download.
if not (ROOT / "libmongocrypt").exists():
setup_libmongocrypt()
if not opts.test_min_deps:
UV_ARGS.append(
"--with pymongocrypt@git+https://github.com/mongodb/libmongocrypt@master#subdirectory=bindings/python"
)
# Use the nocrypto build to avoid dependency issues with older windows/python versions.
BASE = ROOT / "libmongocrypt/nocrypto"
if PLATFORM == "linux":
if (BASE / "lib/libmongocrypt.so").exists():
PYMONGOCRYPT_LIB = BASE / "lib/libmongocrypt.so"
else:
PYMONGOCRYPT_LIB = BASE / "lib64/libmongocrypt.so"
elif PLATFORM == "darwin":
PYMONGOCRYPT_LIB = BASE / "lib/libmongocrypt.dylib"
else:
PYMONGOCRYPT_LIB = BASE / "bin/mongocrypt.dll"
if not PYMONGOCRYPT_LIB.exists():
raise RuntimeError("Cannot find libmongocrypt shared object file")
write_env("PYMONGOCRYPT_LIB", PYMONGOCRYPT_LIB.as_posix())
# PATH is updated by configure-env.sh for access to mongocryptd.
if test_name == "encryption":
if not DRIVERS_TOOLS:
raise RuntimeError("Missing DRIVERS_TOOLS")
csfle_dir = Path(f"{DRIVERS_TOOLS}/.evergreen/csfle")
run_command(f"bash {csfle_dir.as_posix()}/setup-secrets.sh", cwd=csfle_dir)
load_config_from_file(csfle_dir / "secrets-export.sh")
run_command(f"bash {csfle_dir.as_posix()}/start-servers.sh")
if sub_test_name == "pyopenssl":
UV_ARGS.append("--extra ocsp")
if opts.crypt_shared:
config = read_env(f"{DRIVERS_TOOLS}/mo-expansion.sh")
CRYPT_SHARED_DIR = Path(config["CRYPT_SHARED_LIB_PATH"]).parent.as_posix()
LOGGER.info("Using crypt_shared_dir %s", CRYPT_SHARED_DIR)
if PLATFORM == "windows":
write_env("PATH", f"{CRYPT_SHARED_DIR}:$PATH")
else:
write_env(
"DYLD_FALLBACK_LIBRARY_PATH",
f"{CRYPT_SHARED_DIR}:${{DYLD_FALLBACK_LIBRARY_PATH:-}}",
)
write_env("LD_LIBRARY_PATH", f"{CRYPT_SHARED_DIR}:${{LD_LIBRARY_PATH:-}}")
if test_name == "kms":
from kms_tester import setup_kms
setup_kms(sub_test_name)
if test_name == "auth_aws" and sub_test_name != "ecs-remote":
auth_aws_dir = f"{DRIVERS_TOOLS}/.evergreen/auth_aws"
if "AWS_ROLE_SESSION_NAME" in os.environ:
write_env("AWS_ROLE_SESSION_NAME")
if sub_test_name != "ecs":
aws_setup = f"{auth_aws_dir}/aws_setup.sh"
run_command(f"bash {aws_setup} {sub_test_name}")
creds = read_env(f"{auth_aws_dir}/test-env.sh")
for name, value in creds.items():
write_env(name, value)
else:
run_command(f"bash {auth_aws_dir}/setup-secrets.sh")
if test_name == "atlas_connect":
secrets = get_secrets("drivers/atlas_connect")
# Write file with Atlas X509 client certificate:
decoded = base64.b64decode(secrets["ATLAS_X509_DEV_CERT_BASE64"]).decode("utf8")
cert_file = ROOT / ".evergreen/atlas_x509_dev_client_certificate.pem"
with cert_file.open("w") as file:
file.write(decoded)
write_env(
"ATLAS_X509_DEV_WITH_CERT",
secrets["ATLAS_X509_DEV"] + "&tlsCertificateKeyFile=" + str(cert_file),
)
# We do not want the default client_context to be initialized.
write_env("DISABLE_CONTEXT")
if test_name == "numpy":
UV_ARGS.append("--with numpy")
if test_name == "perf":
data_dir = ROOT / "specifications/source/benchmarking/data"
if not data_dir.exists():
run_command("git clone --depth 1 https://github.com/mongodb/specifications.git")
run_command("tar xf extended_bson.tgz", cwd=data_dir)
run_command("tar xf parallel.tgz", cwd=data_dir)
run_command("tar xf single_and_multi_document.tgz", cwd=data_dir)
write_env("TEST_PATH", str(data_dir))
write_env("OUTPUT_FILE", str(ROOT / "results.json"))
# Overwrite the UV_PYTHON from the env.sh file.
write_env("UV_PYTHON", "")
UV_ARGS.append(f"--python={PERF_PYTHON_VERSION}")
# PYTHON-4769 Run perf_test.py directly otherwise pytest's test collection negatively
# affects the benchmark results.
if sub_test_name == "sync":
TEST_ARGS = f"test/performance/perf_test.py {TEST_ARGS}"
else:
TEST_ARGS = f"test/performance/async_perf_test.py {TEST_ARGS}"
# Add coverage if requested.
# Only cover CPython. PyPy reports suspiciously low coverage.
if opts.cov and platform.python_implementation() == "CPython":
# Keep in sync with combine-coverage.sh.
# coverage >=5 is needed for relative_files=true.
UV_ARGS.append("--group coverage")
write_env("COVERAGE")
if opts.green_framework:
framework = opts.green_framework or os.environ["GREEN_FRAMEWORK"]
UV_ARGS.append(f"--group {framework}")
if framework == "gevent" and opts.test_min_deps:
# PYTHON-5729. This can be removed when the min supported gevent is moved to 25.9.1.
UV_ARGS.append('--with "setuptools==81.0"')
else:
TEST_ARGS = f"-v --durations=5 {TEST_ARGS}"
TEST_SUITE = TEST_SUITE_MAP.get(test_name)
if TEST_SUITE:
TEST_ARGS = f"-m {TEST_SUITE} {TEST_ARGS}"
write_env("TEST_ARGS", TEST_ARGS)
write_env("UV_ARGS", " ".join(UV_ARGS))
LOGGER.info(f"Setting up test '{test_title}' with {AUTH=} and {SSL=}... done.")
if __name__ == "__main__":
handle_test_env()

View File

@ -1,14 +0,0 @@
#!/bin/bash
# Stop a server that was started using run-mongodb.sh in DRIVERS_TOOLS.
set -eu
HERE=$(dirname ${BASH_SOURCE:-$0})
HERE="$( cd -- "$HERE" > /dev/null 2>&1 && pwd )"
# Try to source the env file.
if [ -f $HERE/env.sh ]; then
echo "Sourcing env file"
source $HERE/env.sh
fi
bash ${DRIVERS_TOOLS}/.evergreen/run-mongodb.sh stop

View File

@ -1,24 +0,0 @@
#!/bin/bash
# Tear down any services that were used by tests.
set -eu
SCRIPT_DIR=$(dirname ${BASH_SOURCE:-$0})
# Try to source the env file.
if [ -f $SCRIPT_DIR/env.sh ]; then
echo "Sourcing env inputs"
. $SCRIPT_DIR/env.sh
else
echo "Not sourcing env inputs"
fi
# Handle test inputs.
if [ -f $SCRIPT_DIR/test-env.sh ]; then
echo "Sourcing test inputs"
. $SCRIPT_DIR/test-env.sh
else
echo "Missing test inputs, please run 'just setup-tests'"
fi
# Teardown the test runner.
uv run $SCRIPT_DIR/teardown_tests.py

View File

@ -1,64 +0,0 @@
from __future__ import annotations
import os
import shutil
import sys
from pathlib import Path
from utils import DRIVERS_TOOLS, LOGGER, ROOT, run_command
TEST_NAME = os.environ.get("TEST_NAME", "unconfigured")
SUB_TEST_NAME = os.environ.get("SUB_TEST_NAME")
LOGGER.info(f"Tearing down tests of type '{TEST_NAME}'...")
# Shut down csfle servers if applicable.
if TEST_NAME == "encryption":
run_command(f"bash {DRIVERS_TOOLS}/.evergreen/csfle/stop-servers.sh")
# Shut down load balancer if applicable.
elif TEST_NAME == "load-balancer":
run_command(f"bash {DRIVERS_TOOLS}/.evergreen/run-load-balancer.sh stop")
# Tear down kms VM if applicable.
elif TEST_NAME == "kms" and SUB_TEST_NAME in ["azure", "gcp"]:
from kms_tester import teardown_kms
teardown_kms(SUB_TEST_NAME)
# Tear down OIDC if applicable.
elif TEST_NAME == "auth_oidc":
from oidc_tester import teardown_oidc
teardown_oidc(SUB_TEST_NAME)
# Tear down ocsp if applicable.
elif TEST_NAME == "ocsp":
run_command(f"bash {DRIVERS_TOOLS}/.evergreen/ocsp/teardown.sh")
# Tear down atlas cluster if applicable.
if TEST_NAME in ["aws_lambda", "search_index"]:
run_command(f"bash {DRIVERS_TOOLS}/.evergreen/atlas/teardown-atlas-cluster.sh")
# Tear down auth_aws if applicable.
# We do not run web-identity hosts on macos, because the hosts lack permissions,
# so there is no reason to run the teardown, which would error with a 401.
elif TEST_NAME == "auth_aws" and sys.platform != "darwin":
run_command(f"bash {DRIVERS_TOOLS}/.evergreen/auth_aws/teardown.sh")
# Tear down perf if applicable.
elif TEST_NAME == "perf":
shutil.rmtree(ROOT / "specifications", ignore_errors=True)
Path(os.environ["OUTPUT_FILE"]).unlink(missing_ok=True)
# Tear down mog_wsgi if applicable.
elif TEST_NAME == "mod_wsgi":
from mod_wsgi_tester import teardown_mod_wsgi
teardown_mod_wsgi()
# Tear down coverage if applicable.
if os.environ.get("COVERAGE"):
shutil.rmtree(".pytest_cache", ignore_errors=True)
LOGGER.info(f"Tearing down tests of type '{TEST_NAME}'... done.")

View File

@ -1,57 +0,0 @@
#!/bin/bash
# shellcheck disable=SC2154
# Upload a coverate report to codecov.
set -eu
HERE=$(dirname ${BASH_SOURCE:-$0})
ROOT=$(dirname "$(dirname $HERE)")
pushd $ROOT > /dev/null
export FNAME=coverage.xml
REQUESTER=${requester:-}
if [ ! -f ".coverage" ]; then
echo "There are no coverage results, not running codecov"
exit 0
fi
if [[ "${REQUESTER}" == "github_pr" || "${REQUESTER}" == "commit" ]]; then
echo "Uploading codecov for $REQUESTER..."
else
echo "Error: requester must be 'github_pr' or 'commit', got '${REQUESTER}'" >&2
exit 1
fi
printf 'sha: %s\n' "$github_commit"
printf 'flag: %s-%s\n' "$build_variant" "$task_name"
printf 'file: %s\n' "$FNAME"
uv tool run --with "coverage[toml]" coverage xml
codecov_args=(
upload-process
--report-type coverage
--disable-search
--fail-on-error
--git-service github
--token "${CODECOV_TOKEN}"
--sha "${github_commit}"
--flag "${build_variant}-${task_name}"
--file "${FNAME}"
)
if [ -n "${github_pr_number:-}" ]; then
printf 'branch: %s:%s\n' "$github_author" "$github_pr_head_branch"
printf 'pr: %s\n' "$github_pr_number"
uv tool run --from codecov-cli codecovcli \
"${codecov_args[@]}" \
--pr "${github_pr_number}" \
--branch "${github_author}:${github_pr_head_branch}"
else
printf 'branch: %s\n' "$branch_name"
uv tool run --from codecov-cli codecovcli \
"${codecov_args[@]}" \
--branch "${branch_name}"
fi
echo "Uploading codecov for $REQUESTER... done."
popd > /dev/null

View File

@ -1,4 +0,0 @@
#!/bin/bash
# Upload a coverate report to s3.
set -eu
aws s3 cp htmlcov/ s3://"$1"/coverage/"$2"/"$3"/htmlcov/ --recursive --acl public-read --region us-east-1

View File

@ -1,228 +0,0 @@
from __future__ import annotations
import argparse
import dataclasses
import logging
import os
import shlex
import subprocess
import sys
from pathlib import Path
from typing import Any
HERE = Path(__file__).absolute().parent
ROOT = HERE.parent.parent
DRIVERS_TOOLS = os.environ.get("DRIVERS_TOOLS", "").replace(os.sep, "/")
TMP_DRIVER_FILE = "/tmp/mongo-python-driver.tgz" # noqa: S108
LOGGER = logging.getLogger("test")
logging.basicConfig(level=logging.INFO, format="%(levelname)-8s %(message)s")
ENV_FILE = HERE / "test-env.sh"
PLATFORM = "windows" if os.name == "nt" else sys.platform.lower()
@dataclasses.dataclass
class Distro:
name: str
version_id: str
arch: str
# Map the test name to a test suite.
TEST_SUITE_MAP = {
"atlas_connect": "atlas_connect",
"auth_aws": "auth_aws",
"auth_oidc": "auth_oidc",
"default": "",
"default_async": "default_async",
"default_sync": "default",
"encryption": "encryption",
"enterprise_auth": "auth",
"search_index": "search_index",
"kms": "kms",
"load_balancer": "load_balancer",
"mockupdb": "mockupdb",
"ocsp": "ocsp",
"perf": "perf",
"numpy": "",
}
# Tests that require a sub test suite.
SUB_TEST_REQUIRED = ["auth_aws", "auth_oidc", "kms", "mod_wsgi", "perf"]
EXTRA_TESTS = ["mod_wsgi", "aws_lambda", "doctest"]
# Tests that do not use run-mongodb directly.
NO_RUN_ORCHESTRATION = [
"auth_oidc",
"atlas_connect",
"aws_lambda",
"mockupdb",
"ocsp",
]
# Mapping of env variables to options
OPTION_TO_ENV_VAR = {"cov": "COVERAGE", "crypt_shared": "TEST_CRYPT_SHARED"}
def get_test_options(
description, require_sub_test_name=True, allow_extra_opts=False
) -> tuple[argparse.Namespace, list[str]]:
parser = argparse.ArgumentParser(
description=description, formatter_class=argparse.RawDescriptionHelpFormatter
)
if require_sub_test_name:
parser.add_argument(
"test_name",
choices=sorted(list(TEST_SUITE_MAP) + EXTRA_TESTS),
nargs="?",
default="default",
help="The optional name of the test suite to set up, typically the same name as a pytest marker.",
)
parser.add_argument(
"sub_test_name", nargs="?", help="The optional sub test name, for example 'azure'."
)
else:
parser.add_argument(
"test_name",
choices=set(list(TEST_SUITE_MAP) + EXTRA_TESTS) - set(NO_RUN_ORCHESTRATION),
nargs="?",
default="default",
help="The optional name of the test suite to be run, which informs the server configuration.",
)
parser.add_argument(
"--verbose", "-v", action="store_true", help="Whether to log at the DEBUG level."
)
parser.add_argument(
"--quiet", "-q", action="store_true", help="Whether to log at the WARNING level."
)
parser.add_argument("--auth", action="store_true", help="Whether to add authentication.")
parser.add_argument("--ssl", action="store_true", help="Whether to add TLS configuration.")
parser.add_argument(
"--test-min-deps", action="store_true", help="Test against minimum dependency versions"
)
# Add the test modifiers.
if require_sub_test_name:
parser.add_argument(
"--debug-log", action="store_true", help="Enable pymongo standard logging."
)
parser.add_argument("--cov", action="store_true", help="Add test coverage.")
parser.add_argument(
"--green-framework",
nargs=1,
choices=["gevent"],
help="Optional green framework to test against.",
)
parser.add_argument(
"--compressor",
nargs=1,
choices=["zlib", "zstd", "snappy"],
help="Optional compression algorithm.",
)
parser.add_argument("--crypt-shared", action="store_true", help="Test with crypt_shared.")
parser.add_argument("--no-ext", action="store_true", help="Run without c extensions.")
parser.add_argument(
"--mongodb-api-version", choices=["1"], help="MongoDB stable API version to use."
)
parser.add_argument(
"--disable-test-commands", action="store_true", help="Disable test commands."
)
# Get the options.
if not allow_extra_opts:
opts, extra_opts = parser.parse_args(), []
else:
opts, extra_opts = parser.parse_known_args()
# Convert list inputs to strings.
for name in vars(opts):
value = getattr(opts, name)
if isinstance(value, list):
setattr(opts, name, value[0])
# Handle validation and environment variable overrides.
test_name = opts.test_name
sub_test_name = opts.sub_test_name if require_sub_test_name else ""
if require_sub_test_name and test_name in SUB_TEST_REQUIRED and not sub_test_name:
raise ValueError(f"Test '{test_name}' requires a sub_test_name")
handle_env_overrides(parser, opts)
if "auth" in test_name:
opts.auth = True
# 'auth_aws ecs' shouldn't have extra auth set.
if test_name == "auth_aws" and sub_test_name == "ecs":
opts.auth = False
if opts.verbose:
LOGGER.setLevel(logging.DEBUG)
elif opts.quiet:
LOGGER.setLevel(logging.WARNING)
return opts, extra_opts
def handle_env_overrides(parser: argparse.ArgumentParser, opts: argparse.Namespace) -> None:
# Get the options, and then allow environment variable overrides.
for key in vars(opts):
if key in OPTION_TO_ENV_VAR:
env_var = OPTION_TO_ENV_VAR[key]
else:
env_var = key.upper()
if env_var in os.environ:
if parser.get_default(key) != getattr(opts, key):
LOGGER.info("Overriding env var '%s' with cli option", env_var)
elif env_var == "AUTH":
opts.auth = os.environ.get("AUTH") == "auth"
elif env_var == "SSL":
ssl_opt = os.environ.get("SSL", "")
opts.ssl = ssl_opt and ssl_opt.lower() != "nossl"
elif isinstance(getattr(opts, key), bool):
if os.environ[env_var]:
setattr(opts, key, True)
else:
setattr(opts, key, os.environ[env_var])
def read_env(path: Path | str) -> dict[str, str]:
config = dict()
with Path(path).open() as fid:
for line in fid.readlines():
if "=" not in line:
continue
name, _, value = line.strip().partition("=")
if value.startswith(('"', "'")):
value = value[1:-1]
name = name.replace("export ", "")
config[name] = value
return config
def write_env(name: str, value: Any = "1") -> None:
with ENV_FILE.open("a", newline="\n") as fid:
# Remove any existing quote chars.
value = str(value).replace('"', "")
fid.write(f'export {name}="{value}"\n')
def run_command(cmd: str | list[str], **kwargs: Any) -> None:
if isinstance(cmd, list):
cmd = " ".join(cmd)
LOGGER.info("Running command '%s'...", cmd)
kwargs.setdefault("check", True)
# Prevent overriding the python used by other tools.
env = kwargs.pop("env", os.environ).copy()
if "UV_PYTHON" in env:
del env["UV_PYTHON"]
kwargs["env"] = env
try:
subprocess.run(shlex.split(cmd), **kwargs) # noqa: PLW1510, S603
except subprocess.CalledProcessError as e:
LOGGER.error(e.output)
LOGGER.error(str(e))
sys.exit(e.returncode)
LOGGER.info("Running command '%s'... done.", cmd)
def create_archive() -> str:
run_command("git add .", cwd=ROOT)
run_command('git commit --no-verify -m "add files"', check=False, cwd=ROOT)
run_command(f"git archive -o {TMP_DRIVER_FILE} HEAD", cwd=ROOT)
return TMP_DRIVER_FILE

View File

@ -1,18 +0,0 @@
#!/bin/bash
# Set up a remote evergreen spawn host.
set -eu
if [ -z "$1" ]
then
echo "Must supply a spawn host URL!"
fi
target=$1
user=${target%@*}
remote_dir=/home/$user/mongo-python-driver
echo "Copying files to $target..."
rsync -az -e ssh --exclude '.git' --filter=':- .gitignore' -r . $target:$remote_dir
echo "Copying files to $target... done"
ssh $target "$remote_dir/.evergreen/scripts/setup-system.sh"

View File

@ -1,24 +0,0 @@
diff --git a/test/connection_monitoring/pool-create-min-size-error.json b/test/connection_monitoring/pool-create-min-size-error.json
index 1c744b85..509b2a23 100644
--- a/test/connection_monitoring/pool-create-min-size-error.json
+++ b/test/connection_monitoring/pool-create-min-size-error.json
@@ -49,15 +49,15 @@
"type": "ConnectionCreated",
"address": 42
},
+ {
+ "type": "ConnectionPoolCleared",
+ "address": 42
+ },
{
"type": "ConnectionClosed",
"address": 42,
"connectionId": 42,
"reason": "error"
- },
- {
- "type": "ConnectionPoolCleared",
- "address": 42
}
],
"ignore": [

View File

@ -1,440 +0,0 @@
diff --git a/test/unified-test-format/invalid/entity-client-observeTracingMessages-additionalProperties.json b/test/unified-test-format/invalid/entity-client-observeTracingMessages-additionalProperties.json
new file mode 100644
index 00000000..aa8046d2
--- /dev/null
+++ b/test/unified-test-format/invalid/entity-client-observeTracingMessages-additionalProperties.json
@@ -0,0 +1,20 @@
+{
+ "description": "entity-client-observeTracingMessages-additionalProperties",
+ "schemaVersion": "1.26",
+ "createEntities": [
+ {
+ "client": {
+ "id": "client0",
+ "observeTracingMessages": {
+ "foo": "bar"
+ }
+ }
+ }
+ ],
+ "tests": [
+ {
+ "description": "observeTracingMessages must not have additional properties'",
+ "operations": []
+ }
+ ]
+}
diff --git a/test/unified-test-format/invalid/entity-client-observeTracingMessages-additionalPropertyType.json b/test/unified-test-format/invalid/entity-client-observeTracingMessages-additionalPropertyType.json
new file mode 100644
index 00000000..0b3a65f5
--- /dev/null
+++ b/test/unified-test-format/invalid/entity-client-observeTracingMessages-additionalPropertyType.json
@@ -0,0 +1,20 @@
+{
+ "description": "entity-client-observeTracingMessages-additionalPropertyType",
+ "schemaVersion": "1.26",
+ "createEntities": [
+ {
+ "client": {
+ "id": "client0",
+ "observeTracingMessages": {
+ "enableCommandPayload": 0
+ }
+ }
+ }
+ ],
+ "tests": [
+ {
+ "description": "observeTracingMessages enableCommandPayload must be boolean",
+ "operations": []
+ }
+ ]
+}
diff --git a/test/unified-test-format/invalid/entity-client-observeTracingMessages-type.json b/test/unified-test-format/invalid/entity-client-observeTracingMessages-type.json
new file mode 100644
index 00000000..de3ef39a
--- /dev/null
+++ b/test/unified-test-format/invalid/entity-client-observeTracingMessages-type.json
@@ -0,0 +1,18 @@
+{
+ "description": "entity-client-observeTracingMessages-type",
+ "schemaVersion": "1.26",
+ "createEntities": [
+ {
+ "client": {
+ "id": "client0",
+ "observeTracingMessages": "foo"
+ }
+ }
+ ],
+ "tests": [
+ {
+ "description": "observeTracingMessages must be an object",
+ "operations": []
+ }
+ ]
+}
diff --git a/test/unified-test-format/invalid/expectedTracingSpans-additionalProperties.json b/test/unified-test-format/invalid/expectedTracingSpans-additionalProperties.json
new file mode 100644
index 00000000..5947a286
--- /dev/null
+++ b/test/unified-test-format/invalid/expectedTracingSpans-additionalProperties.json
@@ -0,0 +1,30 @@
+{
+ "description": "expectedTracingSpans-additionalProperties",
+ "schemaVersion": "1.26",
+ "createEntities": [
+ {
+ "client": {
+ "id": "client0"
+ }
+ }
+ ],
+ "tests": [
+ {
+ "description": "additional property foo not allowed in expectTracingMessages",
+ "operations": [],
+ "expectTracingMessages": {
+ "client": "client0",
+ "ignoreExtraSpans": false,
+ "spans": [
+ {
+ "name": "command",
+ "tags": {
+ "db.system": "mongodb"
+ }
+ }
+ ],
+ "foo": 0
+ }
+ }
+ ]
+}
diff --git a/test/unified-test-format/invalid/expectedTracingSpans-clientType.json b/test/unified-test-format/invalid/expectedTracingSpans-clientType.json
new file mode 100644
index 00000000..2fe7faea
--- /dev/null
+++ b/test/unified-test-format/invalid/expectedTracingSpans-clientType.json
@@ -0,0 +1,28 @@
+{
+ "description": "expectedTracingSpans-clientType",
+ "schemaVersion": "1.26",
+ "createEntities": [
+ {
+ "client": {
+ "id": "client0"
+ }
+ }
+ ],
+ "tests": [
+ {
+ "description": "client type must be string",
+ "operations": [],
+ "expectTracingMessages": {
+ "client": 0,
+ "spans": [
+ {
+ "name": "command",
+ "tags": {
+ "db.system": "mongodb"
+ }
+ }
+ ]
+ }
+ }
+ ]
+}
diff --git a/test/unified-test-format/invalid/expectedTracingSpans-emptyNestedSpan.json b/test/unified-test-format/invalid/expectedTracingSpans-emptyNestedSpan.json
new file mode 100644
index 00000000..8a98d5ba
--- /dev/null
+++ b/test/unified-test-format/invalid/expectedTracingSpans-emptyNestedSpan.json
@@ -0,0 +1,29 @@
+{
+ "description": "expectedTracingSpans-emptyNestedSpan",
+ "schemaVersion": "1.26",
+ "createEntities": [
+ {
+ "client": {
+ "id": "client0"
+ }
+ }
+ ],
+ "tests": [
+ {
+ "description": "nested spans must not have fewer than 1 items'",
+ "operations": [],
+ "expectTracingMessages": {
+ "client": "client0",
+ "spans": [
+ {
+ "name": "command",
+ "tags": {
+ "db.system": "mongodb"
+ },
+ "nested": []
+ }
+ ]
+ }
+ }
+ ]
+}
diff --git a/test/unified-test-format/invalid/expectedTracingSpans-invalidNestedSpan.json b/test/unified-test-format/invalid/expectedTracingSpans-invalidNestedSpan.json
new file mode 100644
index 00000000..79a86744
--- /dev/null
+++ b/test/unified-test-format/invalid/expectedTracingSpans-invalidNestedSpan.json
@@ -0,0 +1,31 @@
+{
+ "description": "expectedTracingSpans-invalidNestedSpan",
+ "schemaVersion": "1.26",
+ "createEntities": [
+ {
+ "client": {
+ "id": "client0"
+ }
+ }
+ ],
+ "tests": [
+ {
+ "description": "nested span must have required property name",
+ "operations": [],
+ "expectTracingMessages": {
+ "client": "client0",
+ "spans": [
+ {
+ "name": "command",
+ "tags": {
+ "db.system": "mongodb"
+ },
+ "nested": [
+ {}
+ ]
+ }
+ ]
+ }
+ }
+ ]
+}
diff --git a/test/unified-test-format/invalid/expectedTracingSpans-missingPropertyClient.json b/test/unified-test-format/invalid/expectedTracingSpans-missingPropertyClient.json
new file mode 100644
index 00000000..2fb1cd5b
--- /dev/null
+++ b/test/unified-test-format/invalid/expectedTracingSpans-missingPropertyClient.json
@@ -0,0 +1,27 @@
+{
+ "description": "expectedTracingSpans-missingPropertyClient",
+ "schemaVersion": "1.26",
+ "createEntities": [
+ {
+ "client": {
+ "id": "client0"
+ }
+ }
+ ],
+ "tests": [
+ {
+ "description": "missing required property client",
+ "operations": [],
+ "expectTracingMessages": {
+ "spans": [
+ {
+ "name": "command",
+ "tags": {
+ "db.system": "mongodb"
+ }
+ }
+ ]
+ }
+ }
+ ]
+}
diff --git a/test/unified-test-format/invalid/expectedTracingSpans-missingPropertySpans.json b/test/unified-test-format/invalid/expectedTracingSpans-missingPropertySpans.json
new file mode 100644
index 00000000..acd10307
--- /dev/null
+++ b/test/unified-test-format/invalid/expectedTracingSpans-missingPropertySpans.json
@@ -0,0 +1,20 @@
+{
+ "description": "expectedTracingSpans-missingPropertySpans",
+ "schemaVersion": "1.26",
+ "createEntities": [
+ {
+ "client": {
+ "id": "client0"
+ }
+ }
+ ],
+ "tests": [
+ {
+ "description": "missing required property spans",
+ "operations": [],
+ "expectTracingMessages": {
+ "client": "client0"
+ }
+ }
+ ]
+}
diff --git a/test/unified-test-format/invalid/expectedTracingSpans-spanMalformedAdditionalProperties.json b/test/unified-test-format/invalid/expectedTracingSpans-spanMalformedAdditionalProperties.json
new file mode 100644
index 00000000..17299f86
--- /dev/null
+++ b/test/unified-test-format/invalid/expectedTracingSpans-spanMalformedAdditionalProperties.json
@@ -0,0 +1,28 @@
+{
+ "description": "expectedTracingSpans-spanMalformedAdditionalProperties",
+ "schemaVersion": "1.26",
+ "createEntities": [
+ {
+ "client": {
+ "id": "client0"
+ }
+ }
+ ],
+ "tests": [
+ {
+ "description": "Span must not have additional properties",
+ "operations": [],
+ "expectTracingMessages": {
+ "client": "client0",
+ "spans": [
+ {
+ "name": "foo",
+ "tags": {},
+ "nested": [],
+ "foo": "bar"
+ }
+ ]
+ }
+ }
+ ]
+}
diff --git a/test/unified-test-format/invalid/expectedTracingSpans-spanMalformedMissingName.json b/test/unified-test-format/invalid/expectedTracingSpans-spanMalformedMissingName.json
new file mode 100644
index 00000000..0257cd9b
--- /dev/null
+++ b/test/unified-test-format/invalid/expectedTracingSpans-spanMalformedMissingName.json
@@ -0,0 +1,27 @@
+{
+ "description": "expectedTracingSpans-spanMalformedMissingName",
+ "schemaVersion": "1.26",
+ "createEntities": [
+ {
+ "client": {
+ "id": "client0"
+ }
+ }
+ ],
+ "tests": [
+ {
+ "description": "missing required span name",
+ "operations": [],
+ "expectTracingMessages": {
+ "client": "client0",
+ "spans": [
+ {
+ "tags": {
+ "db.system": "mongodb"
+ }
+ }
+ ]
+ }
+ }
+ ]
+}
diff --git a/test/unified-test-format/invalid/expectedTracingSpans-spanMalformedMissingTags.json b/test/unified-test-format/invalid/expectedTracingSpans-spanMalformedMissingTags.json
new file mode 100644
index 00000000..a09ca31c
--- /dev/null
+++ b/test/unified-test-format/invalid/expectedTracingSpans-spanMalformedMissingTags.json
@@ -0,0 +1,25 @@
+{
+ "description": "expectedTracingSpans-spanMalformedMissingTags",
+ "schemaVersion": "1.26",
+ "createEntities": [
+ {
+ "client": {
+ "id": "client0"
+ }
+ }
+ ],
+ "tests": [
+ {
+ "description": "missing required span tags",
+ "operations": [],
+ "expectTracingMessages": {
+ "client": "client0",
+ "spans": [
+ {
+ "name": "foo"
+ }
+ ]
+ }
+ }
+ ]
+}
diff --git a/test/unified-test-format/invalid/expectedTracingSpans-spanMalformedNestedMustBeArray.json b/test/unified-test-format/invalid/expectedTracingSpans-spanMalformedNestedMustBeArray.json
new file mode 100644
index 00000000..ccff0410
--- /dev/null
+++ b/test/unified-test-format/invalid/expectedTracingSpans-spanMalformedNestedMustBeArray.json
@@ -0,0 +1,27 @@
+{
+ "description": "expectedTracingSpans-spanMalformedNestedMustBeArray",
+ "schemaVersion": "1.26",
+ "createEntities": [
+ {
+ "client": {
+ "id": "client0"
+ }
+ }
+ ],
+ "tests": [
+ {
+ "description": "nested spans must be an array",
+ "operations": [],
+ "expectTracingMessages": {
+ "client": "client0",
+ "spans": [
+ {
+ "name": "foo",
+ "tags": {},
+ "nested": {}
+ }
+ ]
+ }
+ }
+ ]
+}
diff --git a/test/unified-test-format/invalid/expectedTracingSpans-spanMalformedTagsMustBeObject.json b/test/unified-test-format/invalid/expectedTracingSpans-spanMalformedTagsMustBeObject.json
new file mode 100644
index 00000000..72af1c29
--- /dev/null
+++ b/test/unified-test-format/invalid/expectedTracingSpans-spanMalformedTagsMustBeObject.json
@@ -0,0 +1,26 @@
+{
+ "description": "expectedTracingSpans-spanMalformedNestedMustBeObject",
+ "schemaVersion": "1.26",
+ "createEntities": [
+ {
+ "client": {
+ "id": "client0"
+ }
+ }
+ ],
+ "tests": [
+ {
+ "description": "span tags must be an object",
+ "operations": [],
+ "expectTracingMessages": {
+ "client": "client0",
+ "spans": [
+ {
+ "name": "foo",
+ "tags": []
+ }
+ ]
+ }
+ }
+ ]
+}

View File

@ -1,26 +0,0 @@
diff --git a/test/auth/legacy/connection-string.json b/test/auth/legacy/connection-string.json
index 3a099c813..8982b61d5 100644
--- a/test/auth/legacy/connection-string.json
+++ b/test/auth/legacy/connection-string.json
@@ -440,6 +440,21 @@
}
}
},
+ {
+ "description": "should throw an exception if username provided (MONGODB-AWS)",
+ "uri": "mongodb://user@localhost.com/?authMechanism=MONGODB-AWS",
+ "valid": false
+ },
+ {
+ "description": "should throw an exception if username and password provided (MONGODB-AWS)",
+ "uri": "mongodb://user:pass@localhost.com/?authMechanism=MONGODB-AWS",
+ "valid": false
+ },
+ {
+ "description": "should throw an exception if AWS_SESSION_TOKEN provided (MONGODB-AWS)",
+ "uri": "mongodb://localhost/?authMechanism=MONGODB-AWS&authMechanismProperties=AWS_SESSION_TOKEN:token",
+ "valid": false
+ },
{
"description": "should recognise the mechanism with test environment (MONGODB-OIDC)",
"uri": "mongodb://localhost/?authMechanism=MONGODB-OIDC&authMechanismProperties=ENVIRONMENT:test",

View File

@ -1,50 +0,0 @@
diff --git a/test/connection_logging/connection-logging.json b/test/connection_logging/connection-logging.json
index 5799e834..72103b3c 100644
--- a/test/connection_logging/connection-logging.json
+++ b/test/connection_logging/connection-logging.json
@@ -446,6 +446,22 @@
}
}
},
+ {
+ "level": "debug",
+ "component": "connection",
+ "data": {
+ "message": "Connection pool cleared",
+ "serverHost": {
+ "$$type": "string"
+ },
+ "serverPort": {
+ "$$type": [
+ "int",
+ "long"
+ ]
+ }
+ }
+ },
{
"level": "debug",
"component": "connection",
@@ -498,22 +514,6 @@
]
}
}
- },
- {
- "level": "debug",
- "component": "connection",
- "data": {
- "message": "Connection pool cleared",
- "serverHost": {
- "$$type": "string"
- },
- "serverPort": {
- "$$type": [
- "int",
- "long"
- ]
- }
- }
}
]
}

View File

@ -1,815 +0,0 @@
diff --git a/test/sessions/snapshot-sessions.json b/test/sessions/snapshot-sessions.json
index 260f8b6f4..8f806ea75 100644
--- a/test/sessions/snapshot-sessions.json
+++ b/test/sessions/snapshot-sessions.json
@@ -988,6 +988,810 @@
}
}
]
+ },
+ {
+ "description": "Find operation with snapshot and snapshot time",
+ "operations": [
+ {
+ "name": "find",
+ "object": "collection0",
+ "arguments": {
+ "session": "session0",
+ "filter": {}
+ },
+ "expectResult": [
+ {
+ "_id": 1,
+ "x": 11
+ },
+ {
+ "_id": 2,
+ "x": 11
+ }
+ ]
+ },
+ {
+ "name": "getSnapshotTime",
+ "object": "session0",
+ "saveResultAsEntity": "savedSnapshotTime"
+ },
+ {
+ "name": "insertOne",
+ "object": "collection0",
+ "arguments": {
+ "document": {
+ "_id": 3,
+ "x": 33
+ }
+ }
+ },
+ {
+ "name": "createEntities",
+ "object": "testRunner",
+ "arguments": {
+ "entities": [
+ {
+ "session": {
+ "id": "session2",
+ "client": "client0",
+ "sessionOptions": {
+ "snapshot": true,
+ "snapshotTime": "savedSnapshotTime"
+ }
+ }
+ }
+ ]
+ }
+ },
+ {
+ "name": "find",
+ "object": "collection0",
+ "arguments": {
+ "session": "session2",
+ "filter": {}
+ },
+ "expectResult": [
+ {
+ "_id": 1,
+ "x": 11
+ },
+ {
+ "_id": 2,
+ "x": 11
+ }
+ ]
+ },
+ {
+ "name": "find",
+ "object": "collection0",
+ "arguments": {
+ "session": "session2",
+ "filter": {}
+ },
+ "expectResult": [
+ {
+ "_id": 1,
+ "x": 11
+ },
+ {
+ "_id": 2,
+ "x": 11
+ }
+ ]
+ },
+ {
+ "name": "find",
+ "object": "collection0",
+ "arguments": {
+ "filter": {}
+ },
+ "expectResult": [
+ {
+ "_id": 1,
+ "x": 11
+ },
+ {
+ "_id": 2,
+ "x": 11
+ },
+ {
+ "_id": 3,
+ "x": 33
+ }
+ ]
+ }
+ ],
+ "expectEvents": [
+ {
+ "client": "client0",
+ "events": [
+ {
+ "commandStartedEvent": {
+ "command": {
+ "find": "collection0",
+ "readConcern": {
+ "level": "snapshot",
+ "atClusterTime": {
+ "$$exists": false
+ }
+ }
+ },
+ "databaseName": "database0"
+ }
+ },
+ {
+ "commandStartedEvent": {
+ "command": {
+ "find": "collection0",
+ "readConcern": {
+ "level": "snapshot",
+ "atClusterTime": {
+ "$$matchesEntity": "savedSnapshotTime"
+ }
+ }
+ },
+ "databaseName": "database0"
+ }
+ },
+ {
+ "commandStartedEvent": {
+ "command": {
+ "find": "collection0",
+ "readConcern": {
+ "level": "snapshot",
+ "atClusterTime": {
+ "$$matchesEntity": "savedSnapshotTime"
+ }
+ }
+ },
+ "databaseName": "database0"
+ }
+ },
+ {
+ "commandStartedEvent": {
+ "command": {
+ "find": "collection0",
+ "readConcern": {
+ "$$exists": false
+ }
+ },
+ "databaseName": "database0"
+ }
+ }
+ ]
+ }
+ ]
+ },
+ {
+ "description": "Distinct operation with snapshot and snapshot time",
+ "operations": [
+ {
+ "name": "distinct",
+ "object": "collection0",
+ "arguments": {
+ "session": "session0",
+ "filter": {},
+ "fieldName": "x"
+ },
+ "expectResult": [
+ 11
+ ]
+ },
+ {
+ "name": "getSnapshotTime",
+ "object": "session0",
+ "saveResultAsEntity": "savedSnapshotTime"
+ },
+ {
+ "name": "insertOne",
+ "object": "collection0",
+ "arguments": {
+ "document": {
+ "_id": 3,
+ "x": 33
+ }
+ }
+ },
+ {
+ "name": "createEntities",
+ "object": "testRunner",
+ "arguments": {
+ "entities": [
+ {
+ "session": {
+ "id": "session2",
+ "client": "client0",
+ "sessionOptions": {
+ "snapshot": true,
+ "snapshotTime": "savedSnapshotTime"
+ }
+ }
+ }
+ ]
+ }
+ },
+ {
+ "name": "distinct",
+ "object": "collection0",
+ "arguments": {
+ "session": "session2",
+ "filter": {},
+ "fieldName": "x"
+ },
+ "expectResult": [
+ 11
+ ]
+ },
+ {
+ "name": "distinct",
+ "object": "collection0",
+ "arguments": {
+ "session": "session2",
+ "filter": {},
+ "fieldName": "x"
+ },
+ "expectResult": [
+ 11
+ ]
+ },
+ {
+ "name": "distinct",
+ "object": "collection0",
+ "arguments": {
+ "filter": {},
+ "fieldName": "x"
+ },
+ "expectResult": [
+ 11,
+ 33
+ ]
+ }
+ ],
+ "expectEvents": [
+ {
+ "client": "client0",
+ "events": [
+ {
+ "commandStartedEvent": {
+ "command": {
+ "distinct": "collection0",
+ "readConcern": {
+ "level": "snapshot",
+ "atClusterTime": {
+ "$$exists": false
+ }
+ }
+ },
+ "databaseName": "database0"
+ }
+ },
+ {
+ "commandStartedEvent": {
+ "command": {
+ "distinct": "collection0",
+ "readConcern": {
+ "level": "snapshot",
+ "atClusterTime": {
+ "$$matchesEntity": "savedSnapshotTime"
+ }
+ }
+ },
+ "databaseName": "database0"
+ }
+ },
+ {
+ "commandStartedEvent": {
+ "command": {
+ "distinct": "collection0",
+ "readConcern": {
+ "level": "snapshot",
+ "atClusterTime": {
+ "$$matchesEntity": "savedSnapshotTime"
+ }
+ }
+ },
+ "databaseName": "database0"
+ }
+ },
+ {
+ "commandStartedEvent": {
+ "command": {
+ "distinct": "collection0",
+ "readConcern": {
+ "$$exists": false
+ }
+ },
+ "databaseName": "database0"
+ }
+ }
+ ]
+ }
+ ]
+ },
+ {
+ "description": "Aggregate operation with snapshot and snapshot time",
+ "operations": [
+ {
+ "name": "aggregate",
+ "object": "collection0",
+ "arguments": {
+ "session": "session0",
+ "pipeline": [
+ {
+ "$match": {
+ "_id": 1
+ }
+ }
+ ]
+ },
+ "expectResult": [
+ {
+ "_id": 1,
+ "x": 11
+ }
+ ]
+ },
+ {
+ "name": "getSnapshotTime",
+ "object": "session0",
+ "saveResultAsEntity": "savedSnapshotTime"
+ },
+ {
+ "name": "findOneAndUpdate",
+ "object": "collection0",
+ "arguments": {
+ "filter": {
+ "_id": 1
+ },
+ "update": {
+ "$inc": {
+ "x": 1
+ }
+ },
+ "returnDocument": "After"
+ },
+ "expectResult": {
+ "_id": 1,
+ "x": 12
+ }
+ },
+ {
+ "name": "createEntities",
+ "object": "testRunner",
+ "arguments": {
+ "entities": [
+ {
+ "session": {
+ "id": "session2",
+ "client": "client0",
+ "sessionOptions": {
+ "snapshot": true,
+ "snapshotTime": "savedSnapshotTime"
+ }
+ }
+ }
+ ]
+ }
+ },
+ {
+ "name": "aggregate",
+ "object": "collection0",
+ "arguments": {
+ "session": "session2",
+ "pipeline": [
+ {
+ "$match": {
+ "_id": 1
+ }
+ }
+ ]
+ },
+ "expectResult": [
+ {
+ "_id": 1,
+ "x": 11
+ }
+ ]
+ },
+ {
+ "name": "aggregate",
+ "object": "collection0",
+ "arguments": {
+ "session": "session2",
+ "pipeline": [
+ {
+ "$match": {
+ "_id": 1
+ }
+ }
+ ]
+ },
+ "expectResult": [
+ {
+ "_id": 1,
+ "x": 11
+ }
+ ]
+ },
+ {
+ "name": "aggregate",
+ "object": "collection0",
+ "arguments": {
+ "pipeline": [
+ {
+ "$match": {
+ "_id": 1
+ }
+ }
+ ]
+ },
+ "expectResult": [
+ {
+ "_id": 1,
+ "x": 12
+ }
+ ]
+ }
+ ],
+ "expectEvents": [
+ {
+ "client": "client0",
+ "events": [
+ {
+ "commandStartedEvent": {
+ "command": {
+ "aggregate": "collection0",
+ "readConcern": {
+ "level": "snapshot",
+ "atClusterTime": {
+ "$$exists": false
+ }
+ }
+ },
+ "databaseName": "database0"
+ }
+ },
+ {
+ "commandStartedEvent": {
+ "command": {
+ "aggregate": "collection0",
+ "readConcern": {
+ "level": "snapshot",
+ "atClusterTime": {
+ "$$matchesEntity": "savedSnapshotTime"
+ }
+ }
+ },
+ "databaseName": "database0"
+ }
+ },
+ {
+ "commandStartedEvent": {
+ "command": {
+ "aggregate": "collection0",
+ "readConcern": {
+ "level": "snapshot",
+ "atClusterTime": {
+ "$$matchesEntity": "savedSnapshotTime"
+ }
+ }
+ },
+ "databaseName": "database0"
+ }
+ },
+ {
+ "commandStartedEvent": {
+ "command": {
+ "aggregate": "collection0",
+ "readConcern": {
+ "$$exists": false
+ }
+ },
+ "databaseName": "database0"
+ }
+ }
+ ]
+ }
+ ]
+ },
+ {
+ "description": "countDocuments operation with snapshot and snapshot time",
+ "operations": [
+ {
+ "name": "countDocuments",
+ "object": "collection0",
+ "arguments": {
+ "session": "session0",
+ "filter": {}
+ },
+ "expectResult": 2
+ },
+ {
+ "name": "getSnapshotTime",
+ "object": "session0",
+ "saveResultAsEntity": "savedSnapshotTime"
+ },
+ {
+ "name": "insertOne",
+ "object": "collection0",
+ "arguments": {
+ "document": {
+ "_id": 3,
+ "x": 33
+ }
+ }
+ },
+ {
+ "name": "createEntities",
+ "object": "testRunner",
+ "arguments": {
+ "entities": [
+ {
+ "session": {
+ "id": "session2",
+ "client": "client0",
+ "sessionOptions": {
+ "snapshot": true,
+ "snapshotTime": "savedSnapshotTime"
+ }
+ }
+ }
+ ]
+ }
+ },
+ {
+ "name": "countDocuments",
+ "object": "collection0",
+ "arguments": {
+ "session": "session2",
+ "filter": {}
+ },
+ "expectResult": 2
+ },
+ {
+ "name": "countDocuments",
+ "object": "collection0",
+ "arguments": {
+ "session": "session2",
+ "filter": {}
+ },
+ "expectResult": 2
+ },
+ {
+ "name": "countDocuments",
+ "object": "collection0",
+ "arguments": {
+ "filter": {}
+ },
+ "expectResult": 3
+ }
+ ],
+ "expectEvents": [
+ {
+ "client": "client0",
+ "events": [
+ {
+ "commandStartedEvent": {
+ "command": {
+ "aggregate": "collection0",
+ "readConcern": {
+ "level": "snapshot",
+ "atClusterTime": {
+ "$$exists": false
+ }
+ }
+ },
+ "databaseName": "database0"
+ }
+ },
+ {
+ "commandStartedEvent": {
+ "command": {
+ "aggregate": "collection0",
+ "readConcern": {
+ "level": "snapshot",
+ "atClusterTime": {
+ "$$matchesEntity": "savedSnapshotTime"
+ }
+ }
+ },
+ "databaseName": "database0"
+ }
+ },
+ {
+ "commandStartedEvent": {
+ "command": {
+ "aggregate": "collection0",
+ "readConcern": {
+ "level": "snapshot",
+ "atClusterTime": {
+ "$$matchesEntity": "savedSnapshotTime"
+ }
+ }
+ },
+ "databaseName": "database0"
+ }
+ },
+ {
+ "commandStartedEvent": {
+ "command": {
+ "aggregate": "collection0",
+ "readConcern": {
+ "$$exists": false
+ }
+ },
+ "databaseName": "database0"
+ }
+ }
+ ]
+ }
+ ]
+ },
+ {
+ "description": "Mixed operation with snapshot and snapshotTime",
+ "operations": [
+ {
+ "name": "find",
+ "object": "collection0",
+ "arguments": {
+ "session": "session0",
+ "filter": {
+ "_id": 1
+ }
+ },
+ "expectResult": [
+ {
+ "_id": 1,
+ "x": 11
+ }
+ ]
+ },
+ {
+ "name": "getSnapshotTime",
+ "object": "session0",
+ "saveResultAsEntity": "savedSnapshotTime"
+ },
+ {
+ "name": "findOneAndUpdate",
+ "object": "collection0",
+ "arguments": {
+ "filter": {
+ "_id": 1
+ },
+ "update": {
+ "$inc": {
+ "x": 1
+ }
+ },
+ "returnDocument": "After"
+ },
+ "expectResult": {
+ "_id": 1,
+ "x": 12
+ }
+ },
+ {
+ "name": "createEntities",
+ "object": "testRunner",
+ "arguments": {
+ "entities": [
+ {
+ "session": {
+ "id": "session2",
+ "client": "client0",
+ "sessionOptions": {
+ "snapshot": true,
+ "snapshotTime": "savedSnapshotTime"
+ }
+ }
+ }
+ ]
+ }
+ },
+ {
+ "name": "find",
+ "object": "collection0",
+ "arguments": {
+ "filter": {
+ "_id": 1
+ }
+ },
+ "expectResult": [
+ {
+ "_id": 1,
+ "x": 12
+ }
+ ]
+ },
+ {
+ "name": "aggregate",
+ "object": "collection0",
+ "arguments": {
+ "pipeline": [
+ {
+ "$match": {
+ "_id": 1
+ }
+ }
+ ],
+ "session": "session2"
+ },
+ "expectResult": [
+ {
+ "_id": 1,
+ "x": 11
+ }
+ ]
+ },
+ {
+ "name": "distinct",
+ "object": "collection0",
+ "arguments": {
+ "fieldName": "x",
+ "filter": {},
+ "session": "session2"
+ },
+ "expectResult": [
+ 11
+ ]
+ }
+ ],
+ "expectEvents": [
+ {
+ "client": "client0",
+ "events": [
+ {
+ "commandStartedEvent": {
+ "command": {
+ "find": "collection0",
+ "readConcern": {
+ "level": "snapshot",
+ "atClusterTime": {
+ "$$exists": false
+ }
+ }
+ }
+ }
+ },
+ {
+ "commandStartedEvent": {
+ "command": {
+ "find": "collection0",
+ "readConcern": {
+ "$$exists": false
+ }
+ }
+ }
+ },
+ {
+ "commandStartedEvent": {
+ "command": {
+ "aggregate": "collection0",
+ "readConcern": {
+ "level": "snapshot",
+ "atClusterTime": {
+ "$$matchesEntity": "savedSnapshotTime"
+ }
+ }
+ }
+ }
+ },
+ {
+ "commandStartedEvent": {
+ "command": {
+ "distinct": "collection0",
+ "readConcern": {
+ "level": "snapshot",
+ "atClusterTime": {
+ "$$matchesEntity": "savedSnapshotTime"
+ }
+ }
+ }
+ }
+ }
+ ]
+ }
+ ]
}
]
}

View File

@ -1,460 +0,0 @@
diff --git a/test/client-side-encryption/spec/unified/accessToken-azure.json b/test/client-side-encryption/spec/unified/accessToken-azure.json
new file mode 100644
index 00000000..510d8795
--- /dev/null
+++ b/test/client-side-encryption/spec/unified/accessToken-azure.json
@@ -0,0 +1,186 @@
+{
+ "description": "accessToken-azure",
+ "schemaVersion": "1.28",
+ "runOnRequirements": [
+ {
+ "minServerVersion": "4.1.10",
+ "csfle": {
+ "minLibmongocryptVersion": "1.6.0"
+ }
+ }
+ ],
+ "createEntities": [
+ {
+ "client": {
+ "id": "client",
+ "autoEncryptOpts": {
+ "keyVaultNamespace": "keyvault.datakeys",
+ "kmsProviders": {
+ "azure": {
+ "accessToken": {
+ "$$placeholder": 1
+ }
+ }
+ }
+ }
+ }
+ },
+ {
+ "database": {
+ "id": "db",
+ "client": "client",
+ "databaseName": "db"
+ }
+ },
+ {
+ "collection": {
+ "id": "coll",
+ "database": "db",
+ "collectionName": "coll"
+ }
+ },
+ {
+ "clientEncryption": {
+ "id": "clientEncryption",
+ "clientEncryptionOpts": {
+ "keyVaultClient": "client",
+ "keyVaultNamespace": "keyvault.datakeys",
+ "kmsProviders": {
+ "azure": {
+ "accessToken": {
+ "$$placeholder": 1
+ }
+ }
+ }
+ }
+ }
+ }
+ ],
+ "initialData": [
+ {
+ "databaseName": "db",
+ "collectionName": "coll",
+ "documents": [],
+ "createOptions": {
+ "validator": {
+ "$jsonSchema": {
+ "properties": {
+ "secret": {
+ "encrypt": {
+ "keyId": [
+ {
+ "$binary": {
+ "base64": "AZURE+AAAAAAAAAAAAAAAA==",
+ "subType": "04"
+ }
+ }
+ ],
+ "bsonType": "string",
+ "algorithm": "AEAD_AES_256_CBC_HMAC_SHA_512-Deterministic"
+ }
+ }
+ },
+ "bsonType": "object"
+ }
+ }
+ }
+ },
+ {
+ "databaseName": "keyvault",
+ "collectionName": "datakeys",
+ "documents": [
+ {
+ "_id": {
+ "$binary": {
+ "base64": "AZURE+AAAAAAAAAAAAAAAA==",
+ "subType": "04"
+ }
+ },
+ "keyAltNames": [
+ "my-key"
+ ],
+ "keyMaterial": {
+ "$binary": {
+ "base64": "n+HWZ0ZSVOYA3cvQgP7inN4JSXfOH85IngmeQxRpQHjCCcqT3IFqEWNlrsVHiz3AELimHhX4HKqOLWMUeSIT6emUDDoQX9BAv8DR1+E1w4nGs/NyEneac78EYFkK3JysrFDOgl2ypCCTKAypkn9CkAx1if4cfgQE93LW4kczcyHdGiH36CIxrCDGv1UzAvERN5Qa47DVwsM6a+hWsF2AAAJVnF0wYLLJU07TuRHdMrrphPWXZsFgyV+lRqJ7DDpReKNO8nMPLV/mHqHBHGPGQiRdb9NoJo8CvokGz4+KE8oLwzKf6V24dtwZmRkrsDV4iOhvROAzz+Euo1ypSkL3mw==",
+ "subType": "00"
+ }
+ },
+ "creationDate": {
+ "$date": {
+ "$numberLong": "1552949630483"
+ }
+ },
+ "updateDate": {
+ "$date": {
+ "$numberLong": "1552949630483"
+ }
+ },
+ "status": {
+ "$numberInt": "0"
+ },
+ "masterKey": {
+ "provider": "azure",
+ "keyVaultEndpoint": "key-vault-csfle.vault.azure.net",
+ "keyName": "key-name-csfle"
+ }
+ }
+ ]
+ }
+ ],
+ "tests": [
+ {
+ "description": "Auto encrypt using access token Azure credentials",
+ "operations": [
+ {
+ "name": "insertOne",
+ "arguments": {
+ "document": {
+ "_id": 1,
+ "secret": "string0"
+ }
+ },
+ "object": "coll"
+ }
+ ],
+ "outcome": [
+ {
+ "documents": [
+ {
+ "_id": 1,
+ "secret": {
+ "$binary": {
+ "base64": "AQGVERPgAAAAAAAAAAAAAAAC5DbBSwPwfSlBrDtRuglvNvCXD1KzDuCKY2P+4bRFtHDjpTOE2XuytPAUaAbXf1orsPq59PVZmsbTZbt2CB8qaQ==",
+ "subType": "06"
+ }
+ }
+ }
+ ],
+ "collectionName": "coll",
+ "databaseName": "db"
+ }
+ ]
+ },
+ {
+ "description": "Explicit encrypt using access token Azure credentials",
+ "operations": [
+ {
+ "name": "encrypt",
+ "object": "clientEncryption",
+ "arguments": {
+ "value": "string0",
+ "opts": {
+ "keyAltName": "my-key",
+ "algorithm": "AEAD_AES_256_CBC_HMAC_SHA_512-Deterministic"
+ }
+ },
+ "expectResult": {
+ "$binary": {
+ "base64": "AQGVERPgAAAAAAAAAAAAAAAC5DbBSwPwfSlBrDtRuglvNvCXD1KzDuCKY2P+4bRFtHDjpTOE2XuytPAUaAbXf1orsPq59PVZmsbTZbt2CB8qaQ==",
+ "subType": "06"
+ }
+ }
+ }
+ ]
+ }
+ ]
+}
diff --git a/test/client-side-encryption/spec/unified/accessToken-gcp.json b/test/client-side-encryption/spec/unified/accessToken-gcp.json
new file mode 100644
index 00000000..f5cf8914
--- /dev/null
+++ b/test/client-side-encryption/spec/unified/accessToken-gcp.json
@@ -0,0 +1,188 @@
+{
+ "description": "accessToken-gcp",
+ "schemaVersion": "1.28",
+ "runOnRequirements": [
+ {
+ "minServerVersion": "4.1.10",
+ "csfle": {
+ "minLibmongocryptVersion": "1.6.0"
+ }
+ }
+ ],
+ "createEntities": [
+ {
+ "client": {
+ "id": "client",
+ "autoEncryptOpts": {
+ "keyVaultNamespace": "keyvault.datakeys",
+ "kmsProviders": {
+ "gcp": {
+ "accessToken": {
+ "$$placeholder": 1
+ }
+ }
+ }
+ }
+ }
+ },
+ {
+ "database": {
+ "id": "db",
+ "client": "client",
+ "databaseName": "db"
+ }
+ },
+ {
+ "collection": {
+ "id": "coll",
+ "database": "db",
+ "collectionName": "coll"
+ }
+ },
+ {
+ "clientEncryption": {
+ "id": "clientEncryption",
+ "clientEncryptionOpts": {
+ "keyVaultClient": "client",
+ "keyVaultNamespace": "keyvault.datakeys",
+ "kmsProviders": {
+ "gcp": {
+ "accessToken": {
+ "$$placeholder": 1
+ }
+ }
+ }
+ }
+ }
+ }
+ ],
+ "initialData": [
+ {
+ "databaseName": "db",
+ "collectionName": "coll",
+ "documents": [],
+ "createOptions": {
+ "validator": {
+ "$jsonSchema": {
+ "properties": {
+ "secret": {
+ "encrypt": {
+ "keyId": [
+ {
+ "$binary": {
+ "base64": "GCP+AAAAAAAAAAAAAAAAAA==",
+ "subType": "04"
+ }
+ }
+ ],
+ "bsonType": "string",
+ "algorithm": "AEAD_AES_256_CBC_HMAC_SHA_512-Deterministic"
+ }
+ }
+ },
+ "bsonType": "object"
+ }
+ }
+ }
+ },
+ {
+ "databaseName": "keyvault",
+ "collectionName": "datakeys",
+ "documents": [
+ {
+ "_id": {
+ "$binary": {
+ "base64": "GCP+AAAAAAAAAAAAAAAAAA==",
+ "subType": "04"
+ }
+ },
+ "keyAltNames": [
+ "my-key"
+ ],
+ "keyMaterial": {
+ "$binary": {
+ "base64": "CiQAIgLj0WyktnB4dfYHo5SLZ41K4ASQrjJUaSzl5vvVH0G12G0SiQEAjlV8XPlbnHDEDFbdTO4QIe8ER2/172U1ouLazG0ysDtFFIlSvWX5ZnZUrRMmp/R2aJkzLXEt/zf8Mn4Lfm+itnjgo5R9K4pmPNvvPKNZX5C16lrPT+aA+rd+zXFSmlMg3i5jnxvTdLHhg3G7Q/Uv1ZIJskKt95bzLoe0tUVzRWMYXLIEcohnQg==",
+ "subType": "00"
+ }
+ },
+ "creationDate": {
+ "$date": {
+ "$numberLong": "1552949630483"
+ }
+ },
+ "updateDate": {
+ "$date": {
+ "$numberLong": "1552949630483"
+ }
+ },
+ "status": {
+ "$numberInt": "0"
+ },
+ "masterKey": {
+ "provider": "gcp",
+ "projectId": "devprod-drivers",
+ "location": "global",
+ "keyRing": "key-ring-csfle",
+ "keyName": "key-name-csfle"
+ }
+ }
+ ]
+ }
+ ],
+ "tests": [
+ {
+ "description": "Auto encrypt using access token GCP credentials",
+ "operations": [
+ {
+ "name": "insertOne",
+ "arguments": {
+ "document": {
+ "_id": 1,
+ "secret": "string0"
+ }
+ },
+ "object": "coll"
+ }
+ ],
+ "outcome": [
+ {
+ "documents": [
+ {
+ "_id": 1,
+ "secret": {
+ "$binary": {
+ "base64": "ARgj/gAAAAAAAAAAAAAAAAACwFd+Y5Ojw45GUXNvbcIpN9YkRdoHDHkR4kssdn0tIMKlDQOLFkWFY9X07IRlXsxPD8DcTiKnl6XINK28vhcGlg==",
+ "subType": "06"
+ }
+ }
+ }
+ ],
+ "collectionName": "coll",
+ "databaseName": "db"
+ }
+ ]
+ },
+ {
+ "description": "Explicit encrypt using access token GCP credentials",
+ "operations": [
+ {
+ "name": "encrypt",
+ "object": "clientEncryption",
+ "arguments": {
+ "value": "string0",
+ "opts": {
+ "keyAltName": "my-key",
+ "algorithm": "AEAD_AES_256_CBC_HMAC_SHA_512-Deterministic"
+ }
+ },
+ "expectResult": {
+ "$binary": {
+ "base64": "ARgj/gAAAAAAAAAAAAAAAAACwFd+Y5Ojw45GUXNvbcIpN9YkRdoHDHkR4kssdn0tIMKlDQOLFkWFY9X07IRlXsxPD8DcTiKnl6XINK28vhcGlg==",
+ "subType": "06"
+ }
+ }
+ }
+ ]
+ }
+ ]
+}
diff --git a/test/unified-test-format/invalid/clientEncryptionOpts-kmsProviders-azure-accessToken-type.json b/test/unified-test-format/invalid/clientEncryptionOpts-kmsProviders-azure-accessToken-type.json
new file mode 100644
index 00000000..8fe5c150
--- /dev/null
+++ b/test/unified-test-format/invalid/clientEncryptionOpts-kmsProviders-azure-accessToken-type.json
@@ -0,0 +1,31 @@
+{
+ "description": "clientEncryptionOpts-kmsProviders-azure-accessToken-type",
+ "schemaVersion": "1.28",
+ "createEntities": [
+ {
+ "client": {
+ "id": "client0"
+ }
+ },
+ {
+ "clientEncryption": {
+ "id": "clientEncryption0",
+ "clientEncryptionOpts": {
+ "keyVaultClient": "client0",
+ "keyVaultNamespace": "keyvault.datakeys",
+ "kmsProviders": {
+ "azure": {
+ "accessToken": 0
+ }
+ }
+ }
+ }
+ }
+ ],
+ "tests": [
+ {
+ "description": "",
+ "operations": []
+ }
+ ]
+}
diff --git a/test/unified-test-format/invalid/clientEncryptionOpts-kmsProviders-gcp-accessToken-type.json b/test/unified-test-format/invalid/clientEncryptionOpts-kmsProviders-gcp-accessToken-type.json
new file mode 100644
index 00000000..2284e26c
--- /dev/null
+++ b/test/unified-test-format/invalid/clientEncryptionOpts-kmsProviders-gcp-accessToken-type.json
@@ -0,0 +1,31 @@
+{
+ "description": "clientEncryptionOpts-kmsProviders-gcp-accessToken-type",
+ "schemaVersion": "1.28",
+ "createEntities": [
+ {
+ "client": {
+ "id": "client0"
+ }
+ },
+ {
+ "clientEncryption": {
+ "id": "clientEncryption0",
+ "clientEncryptionOpts": {
+ "keyVaultClient": "client0",
+ "keyVaultNamespace": "keyvault.datakeys",
+ "kmsProviders": {
+ "gcp": {
+ "accessToken": 0
+ }
+ }
+ }
+ }
+ }
+ ],
+ "tests": [
+ {
+ "description": "",
+ "operations": []
+ }
+ ]
+}

View File

@ -1,20 +0,0 @@
#!/bin/bash
# Synchronize local files to a remote Evergreen spawn host.
set -eu
if [ -z "$1" ]
then
echo "Must supply a spawn host URL!"
fi
target=$1
user=${target%@*}
remote_dir=/home/$user/mongo-python-driver
echo "Copying files to $target..."
rsync -az -e ssh --exclude '.git' --filter=':- .gitignore' -r . $target:$remote_dir
echo "Copying files to $target... done."
echo "Syncing files to $target..."
# shellcheck disable=SC2034
fswatch -o . | while read f; do rsync -hazv -e ssh --exclude '.git' --filter=':- .gitignore' -r . $target:/home/$user/mongo-python-driver; done
echo "Syncing files to $target... done."

View File

@ -1,4 +0,0 @@
# Initial pre-commit reformat
5578999a90e439fbca06fc0ffc98f4d04e96f7b4
# pyupgrade and ruff
0092b0af79378abf35b6db73a082ecb91af1d973

1
.github/CODEOWNERS vendored
View File

@ -1 +0,0 @@
* @mongodb/dbx-python

View File

@ -1,44 +0,0 @@
When reviewing code, focus on:
## Security Critical Issues
- Check for hardcoded secrets, API keys, or credentials.
- Check for instances of potential method call injection, dynamic code execution, symbol injection or other code injection vulnerabilities.
## Performance Red Flags
- Spot inefficient loops and algorithmic issues.
- Check for memory leaks and resource cleanup.
## Code Quality Essentials
- Methods should be focused and appropriately sized. If a method is doing too much, suggest refactorings to split it up.
- Use clear, descriptive naming conventions.
- Avoid encapsulation violations and ensure proper separation of concerns.
- All public classes, modules, and methods should have clear documentation in Sphinx format.
## PyMongo-specific Concerns
- Do not review files within `pymongo/synchronous` or files in `test/` that also have a file of the same name in `test/asynchronous` unless the reviewed changes include a `_IS_SYNC` statement. PyMongo generates these files from `pymongo/asynchronous` and `test/asynchronous` using `tools/synchro.py`.
- All asynchronous functions must not call any blocking I/O.
## Review Style
- Be specific and actionable in feedback.
- Explain the "why" behind recommendations.
- Acknowledge good patterns when you see them.
- Ask clarifying questions when code intent is unclear.
Always prioritize security vulnerabilities and performance issues that could impact users.
Always suggest changes to improve readability and testability. For example, this suggestion seeks to make the code more readable, reusable, and testable:
```python
# Instead of:
if user.email and "@" in user.email and len(user.email) > 5:
submit_button.enabled = True
else:
submit_button.enabled = False
# Consider:
def valid_email(email):
return email and "@" in email and len(email) > 5
submit_button.enabled = valid_email(user.email)
```

View File

@ -1,18 +0,0 @@
version: 2
updates:
# GitHub Actions
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "weekly"
cooldown:
default-days: 7
groups:
actions:
patterns:
- "*"
# Python
- package-ecosystem: "pip"
directory: "/"
schedule:
interval: "weekly"

View File

@ -1,33 +0,0 @@
<!-- Thanks for contributing! -->
<!-- Please ensure that the title of the PR is in the following form:
[JIRA TICKET]: Issue Title
If you are an external contributor and there is no JIRA ticket associated with your change, then use your best judgement
for the PR title. A MongoDB employee will create a JIRA ticket and edit the name and links as appropriate.
Note on AI Contributions:
We only accept pull requests that are authored and submitted by human contributors who fully understand the changes they are proposing.
All contributions must be written and understood by human contributors. Please read about our policy in our contributing guide.
-->
[JIRA TICKET]
## Changes in this PR
<!-- What changes did you make to the code? What new APIs (public or private) were added, removed, or edited to generate
the desired outcome explained in the above summary? -->
## Test Plan
<!-- How did you test the code? If you added unit tests, you can say that. If you didnt introduce unit tests, explain why.
All code should be tested in some way so please list what your validation strategy was. -->
## Checklist
<!-- Do not delete the items provided on this checklist. -->
### Checklist for Author
- [ ] Did you update the changelog (if necessary)?
- [ ] Is there test coverage?
- [ ] Is any followup work tracked in a JIRA ticket? If so, add link(s).
### Checklist for Reviewer
- [ ] Does the title of the PR reference a JIRA Ticket?
- [ ] Do you fully understand the implementation? (Would you be comfortable explaining how this code works to someone else?)
- [ ] Is all relevant documentation (README or docstring) updated?

View File

@ -1,5 +0,0 @@
# List of reviewers for auto-assignment of reviews.
caseyclements
blink1073
Jibola
NoahStapp

View File

@ -1,68 +0,0 @@
name: "CodeQL"
on:
push:
branches: [ "master", "v*"]
tags: ['*']
pull_request:
workflow_call:
inputs:
ref:
required: true
type: string
schedule:
- cron: '17 10 * * 2'
concurrency:
group: codeql-${{ github.ref }}
cancel-in-progress: true
jobs:
analyze:
name: Analyze (${{ matrix.language }})
runs-on: "ubuntu-latest"
timeout-minutes: 360
permissions:
# required for all workflows
security-events: write
strategy:
fail-fast: false
matrix:
include:
- language: c-cpp
build-mode: manual
- language: python
build-mode: none
- language: actions
build-mode: none
steps:
- name: Checkout repository
uses: actions/checkout@v6
with:
ref: ${{ inputs.ref }}
persist-credentials: false
- uses: actions/setup-python@v6
# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
uses: github/codeql-action/init@5d4e8d1aca955e8d8589aabd499c5cae939e33c7 # v4
with:
languages: ${{ matrix.language }}
build-mode: ${{ matrix.build-mode }}
# For more details on CodeQL's query packs, refer to: https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#using-queries-in-ql-packs
queries: security-extended
config: |
paths-ignore:
- 'doc/**'
- 'tools/**'
- 'test/**'
- if: matrix.build-mode == 'manual'
run: |
pip install -e .
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@5d4e8d1aca955e8d8589aabd499c5cae939e33c7 # v4
with:
category: "/language:${{matrix.language}}"

View File

@ -1,57 +0,0 @@
name: Create Release Branch
on:
workflow_dispatch:
inputs:
branch_name:
description: The name of the new branch
required: true
version:
description: The version to set on the branch
required: true
base_ref:
description: The base reference for the branch
push_changes:
description: Whether to push the changes
default: "true"
concurrency:
group: create-branch-${{ github.ref }}
cancel-in-progress: true
defaults:
run:
shell: bash -eux {0}
jobs:
create-branch:
environment: release
runs-on: ubuntu-latest
permissions:
id-token: write
contents: write
outputs:
version: ${{ steps.pre-publish.outputs.version }}
steps:
- uses: mongodb-labs/drivers-github-tools/secure-checkout@v3
with:
app_id: ${{ vars.APP_ID }}
private_key: ${{ secrets.APP_PRIVATE_KEY }}
- uses: mongodb-labs/drivers-github-tools/setup@v3
with:
aws_role_arn: ${{ secrets.AWS_ROLE_ARN }}
aws_region_name: ${{ vars.AWS_REGION_NAME }}
aws_secret_id: ${{ secrets.AWS_SECRET_ID }}
artifactory_username: ${{ vars.ARTIFACTORY_USERNAME }}
- name: Get hatch
run: pip install hatch
- uses: mongodb-labs/drivers-github-tools/create-branch@v3
id: create-branch
with:
branch_name: ${{ inputs.branch_name }}
version: ${{ inputs.version }}
base_ref: ${{ inputs.base_ref }}
push_changes: ${{ inputs.push_changes }}
version_bump_script: hatch version
evergreen_project: mongo-python-driver-release
release_workflow_path: ./.github/workflows/release-python.yml

View File

@ -1,148 +0,0 @@
name: Python Dist
on:
push:
tags:
- "[0-9]+.[0-9]+.[0-9]+"
- "[0-9]+.[0-9]+.[0-9]+.post[0-9]+"
- "[0-9]+.[0-9]+.[0-9]+[a-b][0-9]+"
- "[0-9]+.[0-9]+.[0-9]+rc[0-9]+"
workflow_dispatch:
pull_request:
workflow_call:
inputs:
ref:
required: true
type: string
concurrency:
group: dist-${{ github.ref }}
cancel-in-progress: true
defaults:
run:
shell: bash -eux {0}
jobs:
build_wheels:
name: Build wheels for ${{ matrix.buildplat[1] }}
runs-on: ${{ matrix.buildplat[0] }}
strategy:
# Ensure that a wheel builder finishes even if another fails
fail-fast: false
matrix:
# Github Actions doesn't support pairing matrix values together, let's improvise
# https://github.com/github/feedback/discussions/7835#discussioncomment-1769026
buildplat:
- [ubuntu-latest, "manylinux_x86_64", "cp3*-manylinux_x86_64"]
- [ubuntu-latest, "manylinux_aarch64", "cp3*-manylinux_aarch64"]
- [ubuntu-latest, "manylinux_ppc64le", "cp3*-manylinux_ppc64le"]
- [ubuntu-latest, "manylinux_s390x", "cp3*-manylinux_s390x"]
- [ubuntu-latest, "manylinux_i686", "cp3*-manylinux_i686"]
- [windows-2022, "win_amd6", "cp3*-win_amd64"]
- [windows-2022, "win32", "cp3*-win32"]
- [windows-11-arm, "win_arm64", "cp3*-win_arm64"]
- [macos-14, "macos", "cp*-macosx_*"]
steps:
- name: Checkout pymongo
uses: actions/checkout@v6
with:
fetch-depth: 0
persist-credentials: false
ref: ${{ inputs.ref }}
- uses: actions/setup-python@v6
with:
cache: 'pip'
python-version: 3.11
cache-dependency-path: 'pyproject.toml'
allow-prereleases: true
- name: Set up QEMU
if: runner.os == 'Linux'
uses: docker/setup-qemu-action@ce360397dd3f832beb865e1373c09c0e9f86d70a # v4.0.0
with:
# setup-qemu-action by default uses `tonistiigi/binfmt:latest` image,
# which is out of date. This causes seg faults during build.
# Here we manually fix the version.
image: tonistiigi/binfmt:qemu-v8.1.5
platforms: all
- name: Install cibuildwheel
# Note: the default manylinux is manylinux_2_28
run: |
python -m pip install -U pip
python -m pip install "cibuildwheel>=3.2.0,<4"
- name: Build wheels
env:
CIBW_BUILD: ${{ matrix.buildplat[2] }}
run: python -m cibuildwheel --output-dir wheelhouse
- name: Assert all versions in wheelhouse
if: ${{ ! startsWith(matrix.buildplat[1], 'macos') }}
run: |
ls wheelhouse/*cp39*.whl
ls wheelhouse/*cp310*.whl
ls wheelhouse/*cp311*.whl
ls wheelhouse/*cp312*.whl
ls wheelhouse/*cp313*.whl
ls wheelhouse/*cp314*.whl
# Free-threading builds:
ls wheelhouse/*cp314t*.whl
- uses: actions/upload-artifact@v7
with:
name: wheel-${{ matrix.buildplat[1] }}
path: ./wheelhouse/*.whl
if-no-files-found: error
make_sdist:
name: Make SDist
runs-on: macos-latest
steps:
- uses: actions/checkout@v6
with:
fetch-depth: 0
persist-credentials: false
ref: ${{ inputs.ref }}
- uses: actions/setup-python@v6
with:
# Build sdist on lowest supported Python
python-version: "3.9"
- name: Build SDist
run: |
set -ex
python -m pip install -U pip build
python -m build --sdist .
- name: Test SDist
run: |
python -m pip install dist/*.gz
cd ..
python -c "from pymongo import has_c; assert has_c()"
- uses: actions/upload-artifact@v7
with:
name: "sdist"
path: ./dist/*.tar.gz
collect_dist:
runs-on: ubuntu-latest
needs: [build_wheels, make_sdist]
name: Download Wheels
steps:
- name: Download all workflow run artifacts
uses: actions/download-artifact@v8
- name: Flatten directory
working-directory: .
run: |
find . -mindepth 2 -type f -exec mv {} . \;
find . -type d -empty -delete
- uses: actions/upload-artifact@v7
with:
name: all-dist-${{ github.run_id }}
path: "./*"

View File

@ -1,117 +0,0 @@
name: Release
on:
workflow_dispatch:
inputs:
following_version:
description: "The post (dev) version to set"
dry_run:
description: "Dry Run?"
default: false
type: boolean
schedule:
- cron: '30 5 * * *'
env:
# Changes per repo
PRODUCT_NAME: PyMongo
# Changes per branch
EVERGREEN_PROJECT: mongo-python-driver
# Constant
# inputs will be empty on a scheduled run. so, we only set dry_run
# to 'false' when the input is set to 'false'.
DRY_RUN: ${{ ! contains(inputs.dry_run, 'false') }}
FOLLOWING_VERSION: ${{ inputs.following_version || '' }}
defaults:
run:
shell: bash -eux {0}
jobs:
pre-publish:
environment: release
runs-on: ubuntu-latest
if: github.repository_owner == 'mongodb' || github.event_name == 'workflow_dispatch'
permissions:
id-token: write
contents: write
outputs:
version: ${{ steps.pre-publish.outputs.version }}
steps:
- uses: mongodb-labs/drivers-github-tools/secure-checkout@v3
with:
app_id: ${{ vars.APP_ID }}
private_key: ${{ secrets.APP_PRIVATE_KEY }}
- uses: mongodb-labs/drivers-github-tools/setup@v3
with:
aws_role_arn: ${{ secrets.AWS_ROLE_ARN }}
aws_region_name: ${{ vars.AWS_REGION_NAME }}
aws_secret_id: ${{ secrets.AWS_SECRET_ID }}
- uses: mongodb-labs/drivers-github-tools/python/pre-publish@v3
id: pre-publish
with:
dry_run: ${{ env.DRY_RUN }}
build-dist:
needs: [pre-publish]
uses: ./.github/workflows/dist.yml
with:
ref: ${{ needs.pre-publish.outputs.version }}
static-scan:
needs: [pre-publish]
uses: ./.github/workflows/codeql.yml
permissions:
security-events: write
with:
ref: ${{ needs.pre-publish.outputs.version }}
publish:
needs: [build-dist, static-scan]
name: Upload release to PyPI
runs-on: ubuntu-latest
environment: release
permissions:
id-token: write
steps:
- name: Download all the dists
uses: actions/download-artifact@v8
with:
name: all-dist-${{ github.run_id }}
path: dist/
- name: Publish package distributions to TestPyPI
uses: pypa/gh-action-pypi-publish@ed0c53931b1dc9bd32cbe73a98c7f6766f8a527e # release/v1
with:
repository-url: https://test.pypi.org/legacy/
skip-existing: true
attestations: ${{ env.DRY_RUN }}
- name: Publish package distributions to PyPI
if: startsWith(env.DRY_RUN, 'false')
uses: pypa/gh-action-pypi-publish@ed0c53931b1dc9bd32cbe73a98c7f6766f8a527e # release/v1
post-publish:
needs: [publish]
runs-on: ubuntu-latest
environment: release
permissions:
id-token: write
contents: write
attestations: write
security-events: write
steps:
- uses: mongodb-labs/drivers-github-tools/secure-checkout@v3
with:
app_id: ${{ vars.APP_ID }}
private_key: ${{ secrets.APP_PRIVATE_KEY }}
- uses: mongodb-labs/drivers-github-tools/setup@v3
with:
aws_role_arn: ${{ secrets.AWS_ROLE_ARN }}
aws_region_name: ${{ vars.AWS_REGION_NAME }}
aws_secret_id: ${{ secrets.AWS_SECRET_ID }}
- uses: mongodb-labs/drivers-github-tools/python/post-publish@v3
with:
following_version: ${{ env.FOLLOWING_VERSION }}
product_name: ${{ env.PRODUCT_NAME }}
evergreen_project: ${{ env.EVERGREEN_PROJECT }}
token: ${{ github.token }}
dry_run: ${{ env.DRY_RUN }}

View File

@ -1,104 +0,0 @@
name: Generate SBOM
# This workflow uses cyclonedx-py and publishes an sbom.json artifact.
# It runs on manual trigger or when package files change on main branch,
# and creates a PR with the updated SBOM.
# Internal documentation: go/sbom-scope
on:
workflow_dispatch: {}
push:
branches: ['master']
paths:
- 'requirements.txt'
- 'requirements/**.txt'
- '!requirements/docs.txt'
- '!requirements/test.txt'
permissions:
contents: write
pull-requests: write
jobs:
sbom:
name: Generate SBOM and Create PR
runs-on: ubuntu-latest
concurrency:
group: sbom-${{ github.ref }}
cancel-in-progress: false
steps:
- name: Checkout repository
uses: actions/checkout@v6
with:
persist-credentials: false
- name: Set up Python
uses: actions/setup-python@v6
with:
python-version: "3.10"
- name: Generate SBOM
run: |
python -m venv .venv
source .venv/bin/activate
python tools/generate_sbom_requirements.py
pip install -r sbom-requirements.txt
pip install .
pip uninstall -y pip setuptools
deactivate
python -m venv .venv-sbom
source .venv-sbom/bin/activate
pip install cyclonedx-bom==7.2.1
cyclonedx-py environment --spec-version 1.5 --output-format JSON --output-file sbom.json .venv
# Add PURL for pymongo (local package doesn't get PURL automatically)
jq '(.components[] | select(.name == "pymongo" and .purl == null)) |= (. + {purl: ("pkg:pypi/pymongo@" + .version)})' sbom.json > sbom.tmp.json && mv sbom.tmp.json sbom.json
- name: Download CycloneDX CLI
run: |
curl -L -s -o /tmp/cyclonedx "https://github.com/CycloneDX/cyclonedx-cli/releases/download/v0.29.1/cyclonedx-linux-x64"
chmod +x /tmp/cyclonedx
- name: Validate SBOM
run: /tmp/cyclonedx validate --input-file sbom.json --fail-on-errors
- name: Cleanup
if: always()
run: rm -rf .venv .venv-sbom sbom-requirements.txt
- name: Upload SBOM artifact
uses: actions/upload-artifact@v7
with:
name: sbom
path: sbom.json
if-no-files-found: error
- name: Create Pull Request
uses: peter-evans/create-pull-request@c0f553fe549906ede9cf27b5156039d195d2ece0 # v8
with:
token: ${{ secrets.GITHUB_TOKEN }}
commit-message: 'chore: Update SBOM after dependency changes'
branch: auto-update-sbom-${{ github.run_id }}
delete-branch: true
title: 'Automation: Update SBOM'
body: |
## Automated SBOM Update
This PR was automatically generated because dependency manifest files changed.
### Changes
- Updated `sbom.json` to reflect current dependencies
### Verification
The SBOM was generated using cyclonedx-py v7.2.1 with the current Python environment.
### Triggered by
- Commit: ${{ github.sha }}
- Workflow run: ${{ github.run_id }}
---
_This PR was created automatically by the [SBOM workflow](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }})_
labels: |
sbom
automated
dependencies

View File

@ -1,311 +0,0 @@
name: Python Tests
on:
push:
branches: ["master", "v**"]
pull_request:
workflow_dispatch:
concurrency:
group: tests-${{ github.ref }}
cancel-in-progress: true
defaults:
run:
shell: bash -eux {0}
permissions:
contents: read
jobs:
static:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
with:
persist-credentials: false
- name: Install uv
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
with:
enable-cache: true
python-version: "3.10"
- name: Install just
run: uv tool install rust-just
- name: Install Python dependencies
run: |
just install
- name: Run linters
run: |
just lint-manual
- name: Run compilation
run: |
export PYMONGO_C_EXT_MUST_BUILD=1
pip install -v -e .
python tools/fail_if_no_c.py
- name: Run typecheck
run: |
just typing
- run: |
sudo apt-get install -y cppcheck
- run: |
cppcheck --force bson
cppcheck pymongo
build:
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
# Tests currently only pass on ubuntu on GitHub Actions.
os: [ubuntu-latest]
python-version: ["3.10", "pypy-3.11", "3.13t"]
mongodb-version: ["8.0"]
name: CPython ${{ matrix.python-version }}-${{ matrix.os }}
steps:
- uses: actions/checkout@v6
with:
persist-credentials: false
- name: Install uv
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
with:
enable-cache: true
python-version: ${{ matrix.python-version }}
- id: setup-mongodb
uses: mongodb-labs/drivers-evergreen-tools@master
with:
version: "${{ matrix.mongodb-version }}"
- name: Run tests
run: uv run --extra test pytest -v
coverage:
# This enables a coverage report for a given PR, which will be augmented by
# the combined codecov report uploaded in Evergreen.
runs-on: ubuntu-latest
name: Coverage
steps:
- uses: actions/checkout@v6
with:
persist-credentials: false
- name: Install uv
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
with:
enable-cache: true
python-version: "3.10"
- id: setup-mongodb
uses: mongodb-labs/drivers-evergreen-tools@master
with:
version: "8.0"
- name: Install just
run: uv tool install rust-just
- name: Setup tests
run: COVERAGE=1 just setup-tests
- name: Run tests
run: just run-tests
- name: Generate xml report
run: uv tool run --with "coverage[toml]" coverage xml
- name: Upload test results to Codecov
uses: codecov/codecov-action@671740ac38dd9b0130fbe1cec585b89eea48d3de # v5
with:
token: ${{ secrets.CODECOV_TOKEN }}
doctest:
runs-on: ubuntu-latest
name: DocTest
steps:
- uses: actions/checkout@v6
with:
persist-credentials: false
- name: Install uv
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
with:
enable-cache: true
python-version: "3.10"
- name: Install just
run: uv tool install rust-just
- id: setup-mongodb
uses: mongodb-labs/drivers-evergreen-tools@master
with:
version: "8.0"
- name: Install dependencies
run: just install
- name: Run tests
run: |
just setup-tests doctest
just run-tests
docs:
name: Docs Checks
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
with:
persist-credentials: false
- name: Install uv
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
with:
enable-cache: true
python-version: "3.10"
- name: Install just
run: uv tool install rust-just
- name: Install dependencies
run: just install
- name: Build docs
run: just docs
linkcheck:
name: Link Check
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
with:
persist-credentials: false
- name: Install uv
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
with:
enable-cache: true
python-version: "3.10"
- name: Install just
run: uv tool install rust-just
- name: Install dependencies
run: just install
- name: Build docs
run: just docs-linkcheck
typing:
name: Typing Tests
runs-on: ubuntu-latest
strategy:
matrix:
python: ["3.10", "3.11"]
steps:
- uses: actions/checkout@v6
with:
persist-credentials: false
- name: Install uv
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
with:
enable-cache: true
python-version: "${{matrix.python}}"
- name: Install just
run: uv tool install rust-just
- name: Install dependencies
run: |
just install
- name: Run typecheck
run: |
just typing
integration_tests:
runs-on: ubuntu-latest
name: Integration Tests
steps:
- uses: actions/checkout@v6
with:
persist-credentials: false
- name: Install uv
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
with:
enable-cache: true
python-version: "3.10"
- name: Install just
run: uv tool install rust-just
- name: Install dependencies
run: |
just install
- id: setup-mongodb
uses: mongodb-labs/drivers-evergreen-tools@master
- name: Run tests
run: |
just integration-tests
- id: setup-mongodb-ssl
uses: mongodb-labs/drivers-evergreen-tools@master
with:
ssl: true
- name: Run tests
run: |
just integration-tests
make_sdist:
runs-on: ubuntu-latest
name: "Make an sdist"
steps:
- uses: actions/checkout@v6
with:
persist-credentials: false
- uses: actions/setup-python@v6
with:
cache: 'pip'
cache-dependency-path: 'pyproject.toml'
# Build sdist on lowest supported Python
python-version: "3.9"
- name: Build SDist
shell: bash
run: |
pip install build
python -m build --sdist
- uses: actions/upload-artifact@v7
with:
name: "sdist"
path: dist/*.tar.gz
test_sdist:
runs-on: ubuntu-latest
needs: [make_sdist]
name: Install from SDist and Test
timeout-minutes: 20
steps:
- name: Download sdist
uses: actions/download-artifact@v8
with:
path: sdist/
- name: Unpack SDist
shell: bash
run: |
cd sdist
ls
mkdir test
tar --strip-components=1 -zxf *.tar.gz -C ./test
ls test
- uses: actions/setup-python@v6
with:
cache: 'pip'
cache-dependency-path: 'sdist/test/pyproject.toml'
# Test sdist on lowest supported Python
python-version: "3.9"
- id: setup-mongodb
uses: mongodb-labs/drivers-evergreen-tools@master
- name: Run connect test from sdist
shell: bash
run: |
cd sdist/test
ls
which python
pip install -e ".[test]"
PYMONGO_MUST_CONNECT=1 pytest -v -k client_context
test_minimum:
permissions:
contents: read
runs-on: ubuntu-latest
name: Test minimum dependencies and Python
steps:
- uses: actions/checkout@v6
with:
persist-credentials: false
- name: Install uv
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
with:
python-version: "3.9"
- id: setup-mongodb
uses: mongodb-labs/drivers-evergreen-tools@master
with:
version: "8.0"
- name: Run tests
shell: bash
run: |
uv venv
source .venv/bin/activate
uv pip install -e ".[test]" --resolution=lowest-direct --force-reinstall
pytest -v test/test_srv_polling.py test/test_dns.py test/asynchronous/test_srv_polling.py test/asynchronous/test_dns.py

View File

@ -1,21 +0,0 @@
name: GitHub Actions Security Analysis with zizmor 🌈
on:
push:
branches: ["master"]
pull_request:
branches: ["**"]
jobs:
zizmor:
name: zizmor latest via Cargo
runs-on: ubuntu-latest
permissions:
security-events: write
steps:
- name: Checkout repository
uses: actions/checkout@v6
with:
persist-credentials: false
- name: Run zizmor 🌈
uses: zizmorcore/zizmor-action@71321a20a9ded102f6e9ce5718a2fcec2c4f70d8 # v0.5.2

7
.github/zizmor.yml vendored
View File

@ -1,7 +0,0 @@
rules:
unpinned-uses:
config:
policies:
actions/*: ref-pin
mongodb-labs/drivers-github-tools/*: ref-pin
mongodb-labs/drivers-evergreen-tools: ref-pin

35
.gitignore vendored
View File

@ -9,38 +9,9 @@ build/
doc/_build/ doc/_build/
dist/ dist/
tools/settings.py tools/settings.py
drivers-evergreen-tools
pymongo.egg-info/ pymongo.egg-info/
*.so *.so
*.egg* nosetests.xml
setup.cfg
*.egg
.tox .tox
mongocryptd.pid
.idea/
.vscode/
.nova/
.temp/
venv/
secrets-export.sh
libmongocrypt.tar.gz
libmongocrypt/
.venv
expansion.yml
*expansions.yml
.evergreen/scripts/env.sh
.evergreen/scripts/test-env.sh
specifications/
results.json
.evergreen/atlas_x509_dev_client_certificate.pem
# Lambda temp files
test/lambda/.aws-sam
test/lambda/mongodb/pymongo/*
test/lambda/mongodb/gridfs/*
test/lambda/mongodb/bson/*
test/lambda/*.json
# test results and logs
xunit-results/
coverage.xml
server.log
.coverage

View File

@ -1,135 +0,0 @@
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.5.0
hooks:
- id: check-added-large-files
- id: check-case-conflict
- id: check-toml
- id: check-json
- id: check-yaml
exclude: template.yaml
- id: debug-statements
- id: end-of-file-fixer
exclude: WHEEL
exclude_types: [json]
- id: forbid-new-submodules
- id: trailing-whitespace
exclude: .patch
exclude_types: [json]
- repo: https://github.com/astral-sh/ruff-pre-commit
# Ruff version.
rev: v0.1.3
hooks:
- id: ruff
args: ["--fix", "--show-fixes"]
- id: ruff-format
- repo: local
hooks:
- id: synchro
name: synchro
entry: bash ./tools/synchro.sh
language: python
require_serial: true
fail_fast: true
additional_dependencies:
- ruff==0.1.3
- unasync
- repo: https://github.com/adamchainz/blacken-docs
rev: "1.16.0"
hooks:
- id: blacken-docs
additional_dependencies:
- black==22.3.0
- repo: https://github.com/pre-commit/pygrep-hooks
rev: "v1.10.0"
hooks:
- id: rst-backticks
- id: rst-directive-colons
- id: rst-inline-touching-normal
- repo: https://github.com/rstcheck/rstcheck
rev: v6.2.0
hooks:
- id: rstcheck
additional_dependencies: [sphinx]
args: ["--ignore-directives=doctest,testsetup,todo,automodule","--ignore-substitutions=release", "--report-level=error"]
# We use the Python version instead of the original version which seems to require Docker
# https://github.com/koalaman/shellcheck-precommit
- repo: https://github.com/shellcheck-py/shellcheck-py
rev: v0.9.0.6
hooks:
- id: shellcheck
name: shellcheck
args: ["--severity=warning"]
stages: [manual]
- repo: https://github.com/PyCQA/doc8
rev: v1.1.1
hooks:
- id: doc8
args: ["--ignore=D001"] # ignore line length
stages: [manual]
- repo: https://github.com/sirosen/check-jsonschema
rev: 0.29.0
hooks:
- id: check-github-workflows
- id: check-github-actions
- id: check-dependabot
- repo: https://github.com/ariebovenberg/slotscheck
rev: v0.19.0
hooks:
- id: slotscheck
files: \.py$
exclude: "^(test|tools)/"
stages: [manual]
args: ["--no-strict-imports"]
- repo: https://github.com/codespell-project/codespell
rev: "v2.2.6"
hooks:
- id: codespell
# Examples of errors or updates to justify the exceptions:
# - test/test_on_demand_csfle.py:44: FLE ==> FILE
# - test/test_bson.py:1043: fo ==> of, for, to, do, go
# - test/bson_corpus/decimal128-4.json:98: Infinit ==> Infinite
# - test/test_bson.py:267: isnt ==> isn't
# - test/versioned-api/crud-api-version-1-strict.json:514: nin ==> inn, min, bin, nine
# - test/test_client.py:188: te ==> the, be, we, to
args: ["-L", "fle,fo,infinit,isnt,nin,te,aks"]
- repo: local
hooks:
- id: executable-shell
name: executable-shell
entry: chmod +x
language: system
types: [shell]
exclude: |
(?x)(
.evergreen/retry-with-backoff.sh
)
- id: generate-config
name: generate-config
entry: .evergreen/scripts/generate-config.sh
language: python
require_serial: true
additional_dependencies: ["shrub.py>=3.10.0", "pyyaml>=6.0.2"]
- id: uv-lock
name: uv-lock
entry: uv lock
language: python
require_serial: true
files: ^(uv\.lock|pyproject\.toml|requirements.txt|requirements/.*\.txt)$
pass_filenames: false
fail_fast: true
additional_dependencies:
- "uv>=0.8.4"

View File

@ -1,24 +0,0 @@
# .readthedocs.yaml
# Read the Docs configuration file
# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
# Required
version: 2
# Build documentation in the doc/ directory with Sphinx
sphinx:
configuration: doc/conf.py
fail_on_warning: true
# Set the version of Python and requirements required to build the docs.
python:
install:
# Install pymongo itself.
- method: pip
path: .
- requirements: requirements/docs.txt
build:
os: ubuntu-22.04
tools:
python: "3.11"

23
.travis.yml Normal file
View File

@ -0,0 +1,23 @@
language: python
python:
- 2.6
- 2.7
- 3.2
- 3.3
- 3.4
- 3.5
- 3.6
- pypy
- pypy3
services:
- mongodb
script: python setup.py test
install:
# Work around https://github.com/travis-ci/travis-ci/issues/943
- sudo rm -rf /dev/shm && sudo ln -s /run/shm /dev/shm
# Work around https://github.com/travis-ci/travis-ci/issues/5485
- travis_retry pip install setuptools==29.0.1

View File

@ -1,580 +0,0 @@
# Contributing to PyMongo
PyMongo has a large
[community](https://pymongo.readthedocs.io/en/stable/contributors.html)
and contributions are always encouraged. Contributions can be as simple
as minor tweaks to the documentation. Please read these guidelines
before sending a pull request.
## Bugfixes and New Features
Before starting to write code, look for existing
[tickets](https://jira.mongodb.org/browse/PYTHON) or [create
one](https://jira.mongodb.org/browse/PYTHON) for your specific issue or
feature request. That way you avoid working on something that might not
be of interest or that has already been addressed.
## Supported Interpreters
PyMongo supports CPython 3.9+ and PyPy3.9+. Language features not
supported by all interpreters can not be used.
## Style Guide
PyMongo follows [PEP8](http://www.python.org/dev/peps/pep-0008/)
including 4 space indents and 79 character line limits.
## General Guidelines
- Avoid backward breaking changes if at all possible.
- Write inline documentation for new classes and methods.
- We use [uv](https://docs.astral.sh/uv/) for python environment management and packaging.
- We use [just](https://just.systems/man/en/) as our task runner.
- Write tests and make sure they pass (make sure you have a mongod
running on the default port, then execute `just test` from the cmd
line to run the test suite).
- Add yourself to doc/contributors.rst `:)`
## Authoring a Pull Request
**Our Pull Request Policy is based on this** [Code Review Developer
Guide](https://google.github.io/eng-practices/review)
The expectation for any code author is to provide all the context needed
in the space of a pull request for any engineer to feel equipped to
review the code. Depending on the type of change, do your best to
highlight important new functions or objects you've introduced in the
code; think complex functions or new abstractions. Whilst it may seem
like more work for you to adjust your pull request, the reality is your
likelihood for getting review sooner shoots up.
**Self Review Guidelines to follow**
- If the PR is too large, split it if possible.
- Use 250 LoC (excluding test data and config changes) as a
rule-of-thumb.
- Moving and changing code should be in separate PRs or commits.
- Moving: Taking large code blobs and transplanting
them to another file. There\'s generally no (or very
little) actual code changed other than a cut and
paste. It can even be extended to large deletions.
- Changing: Adding code changes (be that refactors or
functionality additions/subtractions).
- These two, when mixed, can muddy understanding and
sometimes make it harder for reviewers to keep track
of things.
- Prefer explaining with code comments instead of PR comments.
**Provide background**
- The PR description and linked tickets should answer the "what" and
"why" of the change. The code change explains the "how".
**Follow the Template**
- Please do not deviate from the template we make; it is there for a
lot of reasons. If it is a one line fix, we still need to have
context on what and why it is needed.
- If making a versioning change, please let that be known. See examples below:
- `versionadded:: 3.11`
- `versionchanged:: 3.5`
### AI-Generated Contributions Policy
#### Our Stance
We only accept pull requests that are authored and submitted by human contributors who fully understand the changes they are proposing. Pull requests that are not clearly owned and understood by a human contributor may be closed. **All contributions must be submitted, reviewed, and understood by human contributors.**
##### Why This Policy Exists
At MongoDB, we understand the power and prevalence of AI tools in software development. With that being said, many MongoDB libraries are foundational tools used in production systems worldwide. The nature of these libraries requires:
- **Deep domain expertise**: MongoDB's wire protocol, BSON specification, connection pooling, authentication mechanisms, and concurrency patterns require an understanding that AI alone cannot substantiate.
- **Long-term maintainability**: Contributors need to be able to explain *why* code is written a certain way, explain design decisions, and be available to iterate on their contributions.
- **Security responsibility**: Authentication, credential handling, and TLS implementation cannot be left to probabilistic code generation.
##### What This Means for Contributors
**Required:**
- Full understanding of every line of code you submit
- Ability to explain and defend your implementation choices
- Willingness to iterate and maintain your contributions
**Encouraged:**
- Using AI assistants as learning tools to understand concepts
- IDE autocomplete features that suggest standard patterns
- AI help for brainstorming approaches (but write the code yourself)
- Writing code using AI tools, reviewing each line and revising code as necessary.
**Not allowed:**
- Submitting PRs generated solely by AI tools
- Copy-pasting AI-generated code without full understanding
##### Disclosure
If you used AI assistance in any way during your contribution, please disclose what the AI assistant was used for in your PR description. We would love to know what tools developers have found useful in iterating in their day to day.
##### Questions?
If you're unsure whether your contribution complies with this policy, please ask for guidance within the scope of the PR and clarify any uncertainty. We're happy to guide contributors toward successful contributions.
---
*This policy helps us maintain the reliability, security, and trustworthiness that production applications depend on. Thank you for understanding and for contributing thoughtfully to PyMongo.*
## Running Linters
PyMongo uses [pre-commit](https://pypi.org/project/pre-commit/) for
managing linting of the codebase. `pre-commit` performs various checks
on all files in PyMongo and uses tools that help follow a consistent
code style within the codebase.
To set up `pre-commit` locally, run:
```bash
brew install pre-commit
pre-commit install
```
To run `pre-commit` manually, run:
```bash
pre-commit run --all-files
```
To run a manual hook like `ruff` manually, run:
```bash
pre-commit run --all-files --hook-stage manual ruff
```
Typically we use `just` to run the linters, e.g.
```bash
just install # this will install a venv with pre-commit installed, and install the pre-commit hook.
just typing-mypy
just run lint-manual
```
## Documentation
To contribute to the [API documentation](https://pymongo.readthedocs.io/en/stable/) just make your
changes to the inline documentation of the appropriate [source code](https://github.com/mongodb/mongo-python-driver) or
[rst file](https://github.com/mongodb/mongo-python-driver/tree/master/doc) in
a branch and submit a [pull request](https://help.github.com/articles/using-pull-requests). You
might also use the GitHub
[Edit](https://github.com/blog/844-forking-with-the-edit-button) button.
We use [reStructuredText](https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html) for all
documentation including narrative docs, and the [Sphinx docstring format](https://sphinx-rtd-tutorial.readthedocs.io/en/latest/docstrings.html).
You can build the documentation locally by running:
```bash
just docs
```
When updating docs, it can be helpful to run the live docs server as:
```bash
just docs-serve
```
Browse to the link provided, and then as you make changes to docstrings or narrative docs,
the pages will re-render and the browser will automatically refresh.
## Running Tests Locally
- Run `just install` to set a local virtual environment, or you can manually
create a virtual environment and run `pytest` directly. If you want to use a specific
version of Python, set `UV_PYTHON` before running `just install`.
- Ensure you have started the appropriate Mongo Server(s). You can run `just run-server` with optional args
to set up the server. All given options will be passed to
[`run-mongodb.sh`](https://github.com/mongodb-labs/drivers-evergreen-tools/blob/master/.evergreen/run-mongodb.sh). Run `$DRIVERS_TOOLS/.evergreen/run-mongodb.sh start -h`
for a full list of options.
- Run `just test` or `pytest` to run all of the tests.
- Append `test/<mod_name>.py::<class_name>::<test_name>` to run
specific tests. You can omit the `<test_name>` to test a full class
and the `<class_name>` to test a full module. For example:
`just test test/test_change_stream.py::TestUnifiedChangeStreamsErrors::test_change_stream_errors_on_ElectionInProgress`.
- Use the `-k` argument to select tests by pattern.
- Run `just test-coverage` to run tests with coverage and display a report. After running tests with coverage, use `just coverage-html` to generate an HTML report in `htmlcov/index.html`.
## Running tests that require secrets, services, or other configuration
### Prerequisites
- Clone `drivers-evergreen-tools`:
`git clone git@github.com:mongodb-labs/drivers-evergreen-tools.git`.
- Run `export DRIVERS_TOOLS=$PWD/drivers-evergreen-tools`. This can be put into a `.bashrc` file
for convenience.
- Some tests require access to [Drivers test secrets](https://github.com/mongodb-labs/drivers-evergreen-tools/tree/master/.evergreen/secrets_handling#secrets-handling).
### Usage
- Run `just run-server` with optional args to set up the server.
- Run `just setup-tests` with optional args to set up the test environment, secrets, etc.
See `just setup-tests -h` for a full list of available options.
- Run `just run-tests` to run the tests in an appropriate Python environment.
- When done, run `just teardown-tests` to clean up and `just stop-server` to stop the server.
### SSL tests
- Run `just run-server --ssl` to start the server with TLS enabled.
- Run `just setup-tests --ssl`.
- Run `just run-tests`.
Note: for general testing purposes with an TLS-enabled server, you can use the following (this should ONLY be used
for local testing):
```python
from pymongo import MongoClient
client = MongoClient(
"mongodb://localhost:27017?tls=true&tlsAllowInvalidCertificates=true"
)
```
If you want to use the actual certificate file then set `tlsCertificateKeyFile` to the local path
to `<repo_roo>/test/certificates/client.pem` and `tlsCAFile` to the local path to `<repo_roo>/test/certificates/ca.pem`.
### Encryption tests
- Run `just run-server` to start the server.
- Run `just setup-tests encryption`.
- Run the tests with `just run-tests`.
To test with `encryption` and `PyOpenSSL`, use `just setup-tests encryption pyopenssl`.
### PyOpenSSL tests
- Run `just run-server` to start the server.
- Run `just setup-tests default_sync pyopenssl`.
- Run the tests with `just run-tests`.
Note: `PyOpenSSL` is not used in async tests, but you can use `just setup-tests default_async pyopenssl`
to verify that PyMongo falls back to the standard library `OpenSSL`.
### Load balancer tests
- Install `haproxy` (available as `brew install haproxy` on macOS).
- Start the server with `just run-server load_balancer`.
- Set up the test with `just setup-tests load_balancer`.
- Run the tests with `just run-tests`.
### AWS auth tests
- Run `just run-server auth_aws` to start the server.
- Run `just setup-tests auth_aws <aws-test-type>` to set up the AWS test.
- Run the tests with `just run-tests`.
### OIDC auth tests
- Run `just setup-tests auth_oidc <oidc-test-type>` to set up the OIDC test.
- Run the tests with `just run-tests`.
The supported types are [`default`, `azure`, `gcp`, `eks`, `aks`, and `gke`].
For the `eks` test, you will need to set up access to the `drivers-test-secrets-role`, see the [Wiki](https://wiki.corp.mongodb.com/spaces/DRIVERS/pages/239737385/Using+AWS+Secrets+Manager+to+Store+Testing+Secrets).
### KMS tests
For KMS tests that are run locally, and expected to fail, in this case using `azure`:
- Run `just run-server`.
- Run `just setup-tests kms azure-fail`.
- Run `just run-tests`.
For KMS tests that run remotely and are expected to pass, in this case using `gcp`:
- Run `just setup-tests kms gcp`.
- Run `just run-tests`.
### Enterprise Auth tests
Note: these tests can only be run from an Evergreen host.
- Run `just run-server enterprise_auth`.
- Run `just setup-tests enterprise_auth`.
- Run `just run-tests`.
### Atlas Connect tests
- Run `just setup-tests atlas_connect`.
- Run `just run-tests`.
### Search Index tests
- Run `just run-server search_index`.
- Run `just setup-tests search_index`.
- Run `just run-tests`.
### MockupDB tests
- Run `just setup-tests mockupdb`.
- Run `just run-tests`.
### Doc tests
The doc tests require a running server.
- Run `just run-server`.
- Run `just setup-tests doctest`.
- Run `just run-tests`.
### Free-threaded Python Tests
In the evergreen builds, the tests are configured to use the free-threaded python from the toolchain.
Locally you can run:
- Run `just run-server`.
- Run `just setup-tests`.
- Run `UV_PYTHON=3.14t just run-tests`.
### AWS Lambda tests
You will need to set up access to the `drivers-test-secrets-role`, see the [Wiki](https://wiki.corp.mongodb.com/spaces/DRIVERS/pages/239737385/Using+AWS+Secrets+Manager+to+Store+Testing+Secrets).
- Run `just setup-tests aws_lambda`.
- Run `just run-tests`.
### mod_wsgi tests
Note: these tests can only be run from an Evergreen Linux host that has the Python toolchain.
- Run `just run-server`.
- Run `just setup-tests mod_wsgi <mode>`.
- Run `just run-tests`.
The `mode` can be `standalone` or `embedded`. For the `replica_set` version of the tests, use
`TOPOLOGY=replica_set just run-server`.
### OCSP tests
- Export the orchestration file, e.g. `export ORCHESTRATION_FILE=rsa-basic-tls-ocsp-disableStapling.json`.
This corresponds to a config file in `$DRIVERS_TOOLS/.evergreen/orchestration/configs/servers`.
MongoDB servers on MacOS and Windows do not staple OCSP responses and only support RSA.
NOTE: because the mock ocsp responder MUST be started prior to the server starting, the ocsp tests start the server
as part of `setup-tests`.
- Run `just setup-tests ocsp <sub test>` (options are "valid", "revoked", "valid-delegate", "revoked-delegate").
- Run `just run-tests`
If you are running one of the `no-responder` tests, omit the `run-server` step.
### Perf Tests
- Start the appropriate server, e.g. `just run-server --version=v8.0-perf --ssl`.
- Set up the tests with `sync` or `async`: `just setup-tests perf sync`.
- Run the tests: `just run-tests`.
## Enable Debug Logs
- Use `-o log_cli_level="DEBUG" -o log_cli=1` with `just test` or `pytest` to output all debug logs to the terminal. **Warning**: This will output a huge amount of logs.
- Add `log_cli=1` and `log_cli_level="DEBUG"` to the `tool.pytest.ini_options` section in `pyproject.toml` to enable debug logs in this manner by default on your machine.
- Set `DEBUG_LOG=1` and run `just setup-tests`, `just-test`, or `pytest` to enable debug logs only for failed tests.
- Finally, you can use `just setup-tests --debug-log`.
- For evergreen patch builds, you can use `evergreen patch --param DEBUG_LOG=1` to enable debug logs for failed tests in the patch.
## Testing minimum dependencies
To run any of the test suites with minimum supported dependencies, pass `--test-min-deps` to
`just setup-tests`.
## Testing time-dependent operations
- `test.utils_shared.delay` - One can trigger an arbitrarily long-running operation on the server using this delay utility
in combination with a `$where` operation. Use this to test behaviors around timeouts or signals.
## Adding a new test suite
- If adding new tests files that should only be run for that test suite, add a pytest marker to the file and add
to the list of pytest markers in `pyproject.toml`. Then add the test suite to the `TEST_SUITE_MAP` in `.evergreen/scripts/utils.py`. If for some reason it is not a pytest-runnable test, add it to the list of `EXTRA_TESTS` instead.
- If the test uses Atlas or otherwise doesn't use `run-mongodb.sh`, add it to the `NO_RUN_ORCHESTRATION` list in
`.evergreen/scripts/utils.py`.
- If there is something special required to run the local server or there is an extra flag that should always be set
like `AUTH`, add that logic to `.evergreen/scripts/run_server.py`.
- The bulk of the logic will typically be in `.evergreen/scripts/setup_tests.py`. This is where you should fetch secrets and make them available using `write_env`, start services, and write other env vars needed using `write_env`.
- If there are any special test considerations, including not running `pytest` at all, handle it in `.evergreen/scripts/run_tests.py`.
- If there are any services or atlas clusters to teardown, handle them in `.evergreen/scripts/teardown_tests.py`.
- Add functions to generate the test variant(s) and task(s) to the `.evergreen/scripts/generate_config.py`.
- There are some considerations about the Python version used in the test:
- If a specific version of Python is needed in a task that is running on variants with a toolchain, use
``TOOLCHAIN_VERSION`` (e.g. `TOOLCHAIN_VERSION=3.10`). The actual path lookup needs to be done on the host, since
tasks are host-agnostic.
- If a specific Python binary is needed (for example on the FIPS host), set `UV_PYTHON=/path/to/python`.
- If a specific Python version is needed and the toolchain will not be available, use `UV_PYTHON` (e.g. `UV_PYTHON=3.11`).
- The default if neither ``TOOLCHAIN_VERSION`` or ``UV_PYTHON`` is set is to use UV to install the minimum
supported version of Python and use that. This ensures a consistent behavior across host types that do not
have the Python toolchain (e.g. Azure VMs), by having a known version of Python with the build headers (`Python.h`)
needed to build the C extensions.
- Regenerate the test variants and tasks using `pre-commit run --all-files generate-config`.
- Make sure to add instructions for running the test suite to `CONTRIBUTING.md`.
## Handling flaky tests
We have a custom `flaky` decorator in [test/asynchronous/utils.py](test/asynchronous/utils.py) that can be used for
tests that are `flaky`. By default the decorator only applies when not running on CPython on Linux, since other
runtimes tend to have more variation. When using the `flaky` decorator, open a corresponding ticket and
a use the ticket number as the "reason" parameter to the decorator, e.g. `@flaky(reason="PYTHON-1234")`.
When running tests locally (not in CI), the `flaky` decorator will be disabled unless `ENABLE_FLAKY` is set.
To disable the `flaky` decorator in CI, you can use `evergreen patch --param DISABLE_FLAKY=1`.
## Integration Tests
The `integration_tests` directory has a set of scripts that verify the usage of PyMongo with downstream packages or frameworks. See the [README](./integration_tests/README.md) for more information.
To run the tests, use `just integration_tests`.
The tests should be able to run with and without SSL enabled.
## Specification Tests
The MongoDB [specifications repository](https://github.com/mongodb/specifications)
holds in progress and completed specifications for features of MongoDB, drivers,
and associated products. PyMongo supports the [Unified Test Format](https://jira.mongodb.org/browse/DRIVERS-709)
for running specification tests to confirm PyMongo behaves as expected.
### Resynchronizing the Specification Tests
If you would like to re-sync the copy of the specification tests in the
PyMongo repository with that which is inside the [specifications
repo](https://github.com/mongodb/specifications), please use the script
provided in `.evergreen/resync-specs.sh`.:
```bash
git clone git@github.com:mongodb/specifications.git
export MDB_SPECS=~/specifications
cd ~/mongo-python-driver/.evergreen
./resync-specs.sh -b "<regex>" spec1 spec2 ...
./resync-specs.sh -b "connection-string*" crud bson-corpus # Updates crud and bson-corpus specs while ignoring all files with the regex "connection-string*"
cd ..
```
The `-b` flag adds as a regex pattern to block files you do not wish to
update in PyMongo. This is primarily helpful if you are implementing a
new feature in PyMongo that has spec tests already implemented, or if
you are attempting to validate new spec tests in PyMongo.
### Automated Specification Test Resyncing
The (`/.evergreen/scripts/resync-all-specs.sh`) script
automatically runs once a week to resync all the specs with the [specifications repo](https://github.com/mongodb/specifications).
A PR will be generated by mongodb-drivers-pr-bot containing any changes picked up by this resync.
The PR description will display the name(s) of the updated specs along
with any errors that occurred.
Spec test changes associated with a behavioral change or bugfix that has yet to be implemented in PyMongo
must be added to a patch file in `/.evergreen/spec-patch`. Each patch
file must be named after the associated PYTHON ticket and contain the
test differences between PyMongo's current tests and the specification.
All changes listed in these patch files will be *undone* by the script and won't
be applied to PyMongo's tests.
When a new test file or folder is added to the spec repo before the associated code changes are implemented, that test's path must be added to `.evergreen/remove-unimplemented-tests.sh` along with a comment indicating the associated PYTHON ticket for those changes.
Any PR that implements a PYTHON ticket documented in a patch file or within `.evergreen/remove-unimplemented-tests.sh` must also remove the associated patch file or entry in `remove-unimplemented-tests.sh`.
#### Adding to a patch file
To add to or create a patch file, run `git diff` to show the desired changes to undo and copy the
results into the patch file.
For example: the imaginary, unimplemented PYTHON-1234 ticket has associated spec test changes. To add those changes to `PYTHON-1234.patch`), do the following:
```bash
git diff HEAD~1 path/to/file >> .evergreen/spec-patch/PYTHON-1234.patch
```
#### Running Locally
Both `resync-all-specs.sh` and `resync-all-specs.py` can be run locally (and won't generate a PR).
```bash
./.evergreen/scripts/resync-all-specs.sh
python3 ./.evergreen/scripts/resync-all-specs.py
```
## Making a Release
Follow the [Python Driver Release Process Wiki](https://wiki.corp.mongodb.com/display/DRIVERS/Python+Driver+Release+Process).
## Project Structure and Asyncio Considerations
This section describes the layout of the `pymongo/` package.
Within `pymongo/`, the code is further divided into the `pymongo/asynchronous` and `pymongo/synchronous` subdirectories.
Files in `pymongo/synchronous` are generated from `pymongo/asynchronous` using the `synchro` pre-commit hook, which uses [unasync](https://github.com/python-trio/unasync/) and some custom transforms.
As a result, **all modifications** within `pymongo` must be made in either the top-level `pymongo` directory when they have to exhibit differing behavior between sync and async contexts or the `pymongo/asynchronous` directory, not `pymongo/synchronous`.
Any changes made directly to files in the `pymongo/synchronous` directory will be overwritten by the `synchro` hook when it is run, which happens automatically on commit.
Some top-level files (e.g. `pymongo/collection.py`) are re-export files for existing import compatibility and should not be modified directly.
The other top-level files (e.g. `pymongo/network_layer.py`, `pymongo/pool_shared.py`) contain either shared code used in both the asynchronous and synchronous APIs, or code that is very different between the two APIs and therefore cannot be generated from the async version using `synchro`.
Run `pre-commit run --all-files synchro` before running tests to generate the latest version of the synchronous code.
To prevent the `synchro` hook from accidentally overwriting code, it first checks to see whether a sync version
of a file is changing and not its async counterpart, and will fail.
In the unlikely scenario that you want to override this behavior, first export `OVERRIDE_SYNCHRO_CHECK=1`.
Sometimes, the `synchro` hook will fail and introduce changes many previously unmodified files. This is due to static
Python errors, such as missing imports, incorrect syntax, or other fatal typos. To resolve these issues,
run `pre-commit run --all-files --hook-stage manual ruff` and fix all reported errors before running the `synchro`
hook again.
## Converting a test to async
The `tools/convert_test_to_async.py` script takes in an existing synchronous test file and outputs a
partially-converted asynchronous version of the same name to the `test/asynchronous` directory.
Use this generated file as a starting point for the completed conversion.
The script is used like so: `python tools/convert_test_to_async.py [test_file.py]`
## CPU profiling
To profile a test script and generate a flame graph, follow these steps:
1. Install `py-spy` if you haven't already:
```bash
pip install py-spy
```
2. Inside your test script, perform any required setup and then loop over the code you want to profile for improved sampling.
3. Run `py-spy record -o <output.svg> -r <sample_rate=100> -- python <path/to/script>` to generate a `.svg` file containing the flame graph.
(Note: on macOS you will need to run this command using `sudo` to allow `py-spy` to attach to the Python process.)
4. If you need to include native code (for example the C extensions), profiling should be done on a Linux system, as macOS and Windows do not support the `--native` option of `py-spy`.
Creating an ubuntu Evergreen spawn host and using `scp` to copy the flamegraph `.svg` file back to your local machine is the best way to do this.
5. You can then view the flamegraph using an SVG viewer like a browser.
## Memory profiling
To test for a memory leak or any memory-related issues, the current best tool is [memray](https://bloomberg.github.io/memray/overview.html).
In order to include code from our C extensions, it must be run in native mode, on Linux.
To do so, either spin up an Ubuntu docker container or an Ubuntu Evergreen spawn host.
From the spawn host or Ubuntu image, do the following:
1. Install `memray` if you haven't already:
```bash
pip install memray
```
2. Inside your test script, perform any required setup and then loop over the code you want to profile for improved sampling.
3. Run memray with the script under test with the `--native` flag, e.g. `python -m memray run --native -o test.bin <path/to/script>`.
4. Generate the flamegraph with `python -m memray flamegraph -o test.html test.bin`.
See the [docs](https://bloomberg.github.io/memray/flamegraph.html) for more options.
5. Then, from the host computer, use either scp or docker cp to copy the flamegraph, e.g. `scp ubuntu@ec2-3-82-52-49.compute-1.amazonaws.com:/home/ubuntu/test.html .`.
6. You can then view the flamegraph html in a browser.
## Dependabot updates
Dependabot will raise PRs at most once per week, grouped by GitHub Actions updates and Python requirement
file updates. We have a pre-commit hook that will update the `uv.lock` file when requirements change.
To update the lock file on a failing PR, you can use a method like `gh pr checkout <pr number>`, then run
`just lint uv-lock` to update the lock file, and then push the changes. If a typing dependency has changed,
also run `just typing` and handle any new findings.

56
CONTRIBUTING.rst Normal file
View File

@ -0,0 +1,56 @@
Contributing to PyMongo
=======================
PyMongo has a large `community
<http://api.mongodb.org/python/current/contributors.html>`_ and
contributions are always encouraged. Contributions can be as simple as
minor tweaks to the documentation. Please read these guidelines before
sending a pull request.
Bugfixes and New Features
-------------------------
Before starting to write code, look for existing `tickets
<https://jira.mongodb.org/browse/PYTHON>`_ or `create one
<https://jira.mongodb.org/browse/PYTHON>`_ for your specific
issue or feature request. That way you avoid working on something
that might not be of interest or that has already been addressed.
Supported Interpreters
----------------------
PyMongo supports CPython 2.4 and newer, PyPy, and Jython. Language
features not supported by all interpreters can not be used (e.g.
the `with statement
<http://docs.python.org/reference/compound_stmts.html#the-with-statement>`_
is not supported in Python 2.4). Please also ensure that your code is
properly converted by `2to3 <http://docs.python.org/library/2to3.html>`_ for
Python 3 support.
Style Guide
-----------
PyMongo follows `PEP8 <http://www.python.org/dev/peps/pep-0008/>`_
including 4 space indents and 79 character line limits.
General Guidelines
------------------
- Avoid backward breaking changes if at all possible.
- Write inline documentation for new classes and methods.
- Write tests and make sure they pass (make sure you have a mongod
running on the default port, then execute ``python setup.py test``
from the cmd line to run the test suite).
- Add yourself to doc/contributors.rst :)
Documentation
-------------
To contribute to the `API documentation <http://api.mongodb.org/python/current/>`_
just make your changes to the inline documentation of the appropriate
`source code <https://github.com/mongodb/mongo-python-driver>`_ or `rst file
<https://github.com/mongodb/mongo-python-driver/tree/master/doc>`_ in a
branch and submit a `pull request <https://help.github.com/articles/using-pull-requests>`_.
You might also use the github `Edit <https://github.com/blog/844-forking-with-the-edit-button>`_
button.

View File

@ -199,3 +199,4 @@
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and See the License for the specific language governing permissions and
limitations under the License. limitations under the License.

10
MANIFEST.in Normal file
View File

@ -0,0 +1,10 @@
include README.rst
include LICENSE
include ez_setup.py
recursive-include doc *.rst
recursive-include doc *.py
recursive-include tools *.py
include tools/README.rst
recursive-include test *.pem
recursive-include test *.py
recursive-include bson *.h

219
README.md
View File

@ -1,219 +0,0 @@
# PyMongo
[![PyPI Version](https://img.shields.io/pypi/v/pymongo)](https://pypi.org/project/pymongo)
[![Python Versions](https://img.shields.io/pypi/pyversions/pymongo)](https://pypi.org/project/pymongo)
[![Monthly Downloads](https://static.pepy.tech/badge/pymongo/month)](https://pepy.tech/project/pymongo)
[![API Documentation Status](https://readthedocs.org/projects/pymongo/badge/?version=stable)](http://pymongo.readthedocs.io/en/stable/api?badge=stable)
[![codecov](https://codecov.io/gh/mongodb/mongo-python-driver/graph/badge.svg?branch=master)](https://codecov.io/gh/mongodb/mongo-python-driver)
## About
The PyMongo distribution contains tools for interacting with MongoDB
database from Python. The `bson` package is an implementation of the
[BSON format](http://bsonspec.org) for Python. The `pymongo` package is
a native Python driver for MongoDB, offering both synchronous and asynchronous APIs. The `gridfs` package is a
[gridfs](https://github.com/mongodb/specifications/blob/master/source/gridfs/gridfs-spec.md/)
implementation on top of `pymongo`.
PyMongo supports MongoDB 4.0, 4.2, 4.4, 5.0, 6.0, 7.0, and 8.0. PyMongo follows [semantic versioning](https://semver.org/spec/v2.0.0.html) for its releases.
## Documentation
Documentation is available at
[mongodb.com](https://www.mongodb.com/docs/languages/python/pymongo-driver/current/).
[API documentation](https://pymongo.readthedocs.io/en/stable/api/) and the [full changelog](https://pymongo.readthedocs.io/en/stable/changelog.html) for each release is available at [readthedocs.io](https://pymongo.readthedocs.io/en/stable/index.html).
## Support / Feedback
For issues with, questions about, or feedback for PyMongo, please look
into our [support channels](https://support.mongodb.com/welcome). Please
do not email any of the PyMongo developers directly with issues or
questions - you're more likely to get an answer on
[StackOverflow](https://stackoverflow.com/questions/tagged/mongodb)
(using a "mongodb" tag).
## Bugs / Feature Requests
Think you've found a bug? Want to see a new feature in PyMongo? Please
open a case in our issue management tool, JIRA:
- [Create an account and login](https://jira.mongodb.org).
- Navigate to [the PYTHON
project](https://jira.mongodb.org/browse/PYTHON).
- Click **Create Issue** - Please provide as much information as
possible about the issue type and how to reproduce it.
Bug reports in JIRA for all driver projects (i.e. PYTHON, CSHARP, JAVA)
and the Core Server (i.e. SERVER) project are **public**.
### How To Ask For Help
Please include all of the following information when opening an issue:
- Detailed steps to reproduce the problem, including full traceback,
if possible.
- The exact python version used, with patch level:
```bash
python -c "import sys; print(sys.version)"
```
- The exact version of PyMongo used, with patch level:
```bash
python -c "import pymongo; print(pymongo.version); print(pymongo.has_c())"
```
- The operating system and version (e.g. Windows 7, OSX 10.8, ...)
- Web framework or asynchronous network library used, if any, with
version (e.g. Django 1.7, mod_wsgi 4.3.0, gevent 1.0.1, Tornado
4.0.2, ...)
### Security Vulnerabilities
If you've identified a security vulnerability in a driver or any other
MongoDB project, please report it according to the [instructions
here](https://www.mongodb.com/docs/manual/tutorial/create-a-vulnerability-report/).
## Installation
PyMongo can be installed with [pip](http://pypi.python.org/pypi/pip):
```bash
python -m pip install pymongo
```
You can also download the project source and do:
```bash
pip install .
```
Do **not** install the "bson" package from pypi. PyMongo comes with
its own bson package; running "pip install bson" installs a third-party
package that is incompatible with PyMongo.
## Dependencies
PyMongo supports CPython 3.9+ and PyPy3.9+.
Required dependencies:
Support for `mongodb+srv://` URIs requires [dnspython](https://pypi.python.org/pypi/dnspython)
Optional dependencies:
GSSAPI authentication requires
[pykerberos](https://pypi.python.org/pypi/pykerberos) on Unix or
[WinKerberos](https://pypi.python.org/pypi/winkerberos) on Windows. The
correct dependency can be installed automatically along with PyMongo:
```bash
python -m pip install "pymongo[gssapi]"
```
MONGODB-AWS authentication requires
[pymongo-auth-aws](https://pypi.org/project/pymongo-auth-aws/):
```bash
python -m pip install "pymongo[aws]"
```
OCSP (Online Certificate Status Protocol) requires
[PyOpenSSL](https://pypi.org/project/pyOpenSSL/),
[requests](https://pypi.org/project/requests/),
[service_identity](https://pypi.org/project/service_identity/) and may
require [certifi](https://pypi.python.org/pypi/certifi):
```bash
python -m pip install "pymongo[ocsp]"
```
Wire protocol compression with snappy requires
[python-snappy](https://pypi.org/project/python-snappy):
```bash
python -m pip install "pymongo[snappy]"
```
Wire protocol compression with zstandard requires
[backports.zstd](https://pypi.org/project/backports.zstd)
when used with Python versions before 3.14:
```bash
python -m pip install "pymongo[zstd]"
```
Client-Side Field Level Encryption requires
[pymongocrypt](https://pypi.org/project/pymongocrypt/) and
[pymongo-auth-aws](https://pypi.org/project/pymongo-auth-aws/):
```bash
python -m pip install "pymongo[encryption]"
```
You can install all dependencies automatically with the following
command:
```bash
python -m pip install "pymongo[gssapi,aws,ocsp,snappy,zstd,encryption]"
```
## Examples
Here's a basic example (for more see the *examples* section of the
docs):
```pycon
>>> import pymongo
>>> client = pymongo.MongoClient("localhost", 27017)
>>> db = client.test
>>> db.name
'test'
>>> db.my_collection
Collection(Database(MongoClient('localhost', 27017), 'test'), 'my_collection')
>>> db.my_collection.insert_one({"x": 10}).inserted_id
ObjectId('4aba15ebe23f6b53b0000000')
>>> db.my_collection.insert_one({"x": 8}).inserted_id
ObjectId('4aba160ee23f6b543e000000')
>>> db.my_collection.insert_one({"x": 11}).inserted_id
ObjectId('4aba160ee23f6b543e000002')
>>> db.my_collection.find_one()
{'x': 10, '_id': ObjectId('4aba15ebe23f6b53b0000000')}
>>> for item in db.my_collection.find():
... print(item["x"])
...
10
8
11
>>> db.my_collection.create_index("x")
'x_1'
>>> for item in db.my_collection.find().sort("x", pymongo.ASCENDING):
... print(item["x"])
...
8
10
11
>>> [item["x"] for item in db.my_collection.find().limit(2).skip(1)]
[8, 11]
```
## Learning Resources
- MongoDB Learn - [Python
courses](https://learn.mongodb.com/catalog?labels=%5B%22Language%22%5D&values=%5B%22Python%22%5D).
- [Python Articles on Developer
Center](https://www.mongodb.com/developer/languages/python/).
## Testing
The easiest way to run the tests is to run the following from the repository root.
```bash
pip install -e ".[test]"
pytest
```
For more advanced testing scenarios, see the [contributing guide](https://github.com/mongodb/mongo-python-driver/blob/master/CONTRIBUTING.md#running-tests-locally).

149
README.rst Normal file
View File

@ -0,0 +1,149 @@
=======
PyMongo
=======
:Info: See `the mongo site <http://www.mongodb.org>`_ for more information. See `github <http://github.com/mongodb/mongo-python-driver/tree>`_ for the latest source.
:Author: Mike Dirolf
:Maintainer: Bernie Hackett <bernie@mongodb.com>
**PyMongo 2.x is in maintenance mode. Support for new MongoDB
features ended with the release of MongoDB 3.0 and PyMongo 2.8. Users are
strongly encouraged to upgrade to PyMongo 3.x.**
About
=====
The PyMongo distribution contains tools for interacting with MongoDB
database from Python. The ``bson`` package is an implementation of
the `BSON format <http://bsonspec.org>`_ for Python. The ``pymongo``
package is a native Python driver for MongoDB. The ``gridfs`` package
is a `gridfs
<http://www.mongodb.org/display/DOCS/GridFS+Specification>`_
implementation on top of ``pymongo``.
Support / Feedback
==================
For issues with, questions about, or feedback for PyMongo, please look into
our `support channels <http://www.mongodb.org/about/support>`_. Please
do not email any of the PyMongo developers directly with issues or
questions - you're more likely to get an answer on the `mongodb-user
<http://groups.google.com/group/mongodb-user>`_ list on Google Groups.
Bugs / Feature Requests
=======================
Think youve found a bug? Want to see a new feature in PyMongo? Please open a
case in our issue management tool, JIRA:
- `Create an account and login <https://jira.mongodb.org>`_.
- Navigate to `the PYTHON project <https://jira.mongodb.org/browse/PYTHON>`_.
- Click **Create Issue** - Please provide as much information as possible about the issue type and how to reproduce it.
Bug reports in JIRA for all driver projects (i.e. PYTHON, CSHARP, JAVA) and the
Core Server (i.e. SERVER) project are **public**.
How To Ask For Help
-------------------
Please include all of the following information when opening an issue:
- Detailed steps to reproduce the problem, including full traceback, if possible.
- The exact python version used, with patch level::
$ python -c "import sys; print(sys.version)"
- The exact version of PyMongo used, with patch level::
$ python -c "import pymongo; print(pymongo.version); print(pymongo.has_c())"
- The operating system and version (e.g. Windows 7, OSX 10.8, ...)
- Web framework or asynchronous network library used, if any, with version (e.g.
Django 1.7, mod_wsgi 4.3.0, gevent 1.0.1, Tornado 4.0.2, ...)
Security Vulnerabilities
------------------------
If youve identified a security vulnerability in a driver or any other
MongoDB project, please report it according to the `instructions here
<http://docs.mongodb.org/manual/tutorial/create-a-vulnerability-report>`_.
Installation
============
If you have `setuptools
<http://pythonhosted.org/setuptools/>`_ installed you
should be able to do **easy_install pymongo** to install
PyMongo. Otherwise you can download the project source and do **python
setup.py install** to install.
Do **not** install the "bson" package. PyMongo comes with its own bson package;
doing "easy_install bson" installs a third-party package that is incompatible
with PyMongo.
Dependencies
============
The PyMongo distribution is supported and tested on Python 2.x (where
x >= 4) and Python 3.x (where x >= 1). PyMongo versions <= 1.3 also
supported Python 2.3, but that is no longer supported.
Additional dependencies are:
- (to generate documentation) sphinx_
- (to auto-discover tests) `nose <http://somethingaboutorange.com/mrl/projects/nose/>`_
Examples
========
Here's a basic example (for more see the *examples* section of the docs):
.. code-block:: pycon
>>> import pymongo
>>> client = pymongo.MongoClient("localhost", 27017)
>>> db = client.test
>>> db.name
u'test'
>>> db.my_collection
Collection(Database(MongoClient('localhost', 27017), u'test'), u'my_collection')
>>> db.my_collection.save({"x": 10})
ObjectId('4aba15ebe23f6b53b0000000')
>>> db.my_collection.save({"x": 8})
ObjectId('4aba160ee23f6b543e000000')
>>> db.my_collection.save({"x": 11})
ObjectId('4aba160ee23f6b543e000002')
>>> db.my_collection.find_one()
{u'x': 10, u'_id': ObjectId('4aba15ebe23f6b53b0000000')}
>>> for item in db.my_collection.find():
... print(item["x"])
...
10
8
11
>>> db.my_collection.create_index("x")
u'x_1'
>>> for item in db.my_collection.find().sort("x", pymongo.ASCENDING):
... print(item["x"])
...
8
10
11
>>> [item["x"] for item in db.my_collection.find().limit(2).skip(1)]
[8, 11]
Documentation
=============
You will need sphinx_ installed to generate the
documentation. Documentation can be generated by running **python
setup.py doc**. Generated documentation can be found in the
*doc/build/html/* directory.
Testing
=======
The easiest way to run the tests is to install `nose
<http://somethingaboutorange.com/mrl/projects/nose/>`_ (**easy_install
nose**) and run **nosetests** or **python setup.py test** in the root
of the distribution. Tests are located in the *test/* directory.
.. _sphinx: http://sphinx.pocoo.org/

70
RELEASE.rst Normal file
View File

@ -0,0 +1,70 @@
Some notes on PyMongo releases
==============================
Versioning
----------
We shoot for a release every few months - that will generally just
increment the middle version number (e.g. 2.1.1 -> 2.2).
Minor releases are reserved for bug fixes (in general no new features
or deprecations) - they only happen in cases where there is a critical
bug in a recently released version, or when a release has no new
features or API changes.
In between releases we use a "+" version number to denote the version
under development. So if we just released 2.1, then the current dev
version would be 2.1+. When we make the next release (2.1.1 or 2.2) we
replace all instances of 2.1+ in the docs with the new version number.
Deprecation
-----------
Changes should be backwards compatible unless absolutely necessary. When making
API changes the approach is generally to add a deprecation warning but keeping
the existing API functional. Eventually (after at least ~4 releases) we can
remove the old API.
Doing a Release
---------------
1. Test release on Python 2.4-2.7 and 3.1-3.3 on Windows, Linux and OSX,
with and without the C extension. Generally enough to just run the tests on
2.4, 2.7 and 3.3 with and without the extension on a single platform,
and then just test any version on the other platforms as a sanity check.
`python setup.py test` will build the extension and test.
`python tools/clean.py` will remove the extension, and then `nosetests` will
run the tests without it. Run the replica set and mongos high-availability
tests with `PYTHONPATH=. python test/high_availability/test_ha.py` and the slow
tests with `nosetests -d test/slow`. Can also run the doctests: `python
setup.py doc -t`. For building extensions on Windows check section below.
2. Add release notes to doc/changelog.rst. Generally just summarize/clarify
the git log, but might add some more long form notes for big changes.
3. Search and replace the "+" version number w/ the new version number (see
note above).
4. Make sure version number is updated in setup.py and pymongo/__init__.py
5. Commit with a BUMP version_number message.
6. Tag w/ version_number
7. Push commit / tag.
8. Push source to PyPI: `python setup.py sdist upload`
9. Push binaries to PyPI; for each version of python and platform do:`python
setup.py bdist_egg upload`. Probably best to do `python setup.py bdist_egg`
first, to make sure the egg builds properly. Notably on the Windows machine,
for Python 2.4 and 2.5, you will have to run `python setup.py build -c mingw32
bdist_egg upload` or the C extension build will fail with an error about Visual
Studio 2003. On Windows we also push a binary installer. The setup.py target
for that is `bdist_wininst`.
10. Make sure the docs have properly updated (driver buildbot does this).
11. Add a "+" to the version number in setup.py/__init__.py, commit, push.
12. Announce!

View File

@ -1,98 +0,0 @@
PyMongo uses third-party libraries or other resources that may
be distributed under licenses different than the PyMongo software.
In the event that we accidentally failed to list a required notice,
please bring it to our attention through any of the ways detailed here:
https://jira.mongodb.org/projects/PYTHON
The attached notices are provided for information only.
For any licenses that require disclosure of source, sources are available at
https://github.com/mongodb/mongo-python-driver.
1) License Notice for time64.c
------------------------------
Copyright (c) 2007-2010 Michael G Schwern
This software originally derived from Paul Sheer's pivotal_gmtime_r.c.
The MIT License:
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
2) License Notice for _asyncio_lock.py
-----------------------------------------
1. This LICENSE AGREEMENT is between the Python Software Foundation
("PSF"), and the Individual or Organization ("Licensee") accessing and
otherwise using this software ("Python") in source or binary form and
its associated documentation.
2. Subject to the terms and conditions of this License Agreement, PSF hereby
grants Licensee a nonexclusive, royalty-free, world-wide license to reproduce,
analyze, test, perform and/or display publicly, prepare derivative works,
distribute, and otherwise use Python alone or in any derivative version,
provided, however, that PSF's License Agreement and PSF's notice of copyright,
i.e., "Copyright (c) 2001-2024 Python Software Foundation; All Rights Reserved"
are retained in Python alone or in any derivative version prepared by Licensee.
3. In the event Licensee prepares a derivative work that is based on
or incorporates Python or any part thereof, and wants to make
the derivative work available to others as provided herein, then
Licensee hereby agrees to include in any such work a brief summary of
the changes made to Python.
4. PSF is making Python available to Licensee on an "AS IS"
basis. PSF MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR
IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, PSF MAKES NO AND
DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS
FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF PYTHON WILL NOT
INFRINGE ANY THIRD PARTY RIGHTS.
5. PSF SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF PYTHON
FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS AS
A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING PYTHON,
OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF.
6. This License Agreement will automatically terminate upon a material
breach of its terms and conditions.
7. Nothing in this License Agreement shall be deemed to create any
relationship of agency, partnership, or joint venture between PSF and
Licensee. This License Agreement does not grant permission to use PSF
trademarks or trade name in a trademark sense to endorse or promote
products or services of Licensee, or any third party.
8. By copying, installing or otherwise using Python, Licensee
agrees to be bound by the terms and conditions of this License
Agreement.
Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH
REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY
AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT,
INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM
LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR
OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
PERFORMANCE OF THIS SOFTWARE.

152
_setup.py
View File

@ -1,152 +0,0 @@
from __future__ import annotations
import os
import sys
import warnings
# Hack to silence atexit traceback in some Python versions
try:
import multiprocessing # noqa: F401
except ImportError:
pass
from setuptools import setup
from setuptools.command.build_ext import build_ext
from setuptools.extension import Extension
class custom_build_ext(build_ext):
"""Allow C extension building to fail.
The C extension speeds up BSON encoding, but is not essential.
"""
warning_message = """
********************************************************************
WARNING: %s could not
be compiled. No C extensions are essential for PyMongo to run,
although they do result in significant speed improvements.
%s
Please see the installation docs for solutions to build issues:
https://pymongo.readthedocs.io/en/stable/installation.html
Here are some hints for popular operating systems:
If you are seeing this message on Linux you probably need to
install GCC and/or the Python development package for your
version of Python.
Debian and Ubuntu users should issue the following command:
$ sudo apt-get install build-essential python-dev
Users of Red Hat based distributions (RHEL, CentOS, Amazon Linux,
Oracle Linux, Fedora, etc.) should issue the following command:
$ sudo yum install gcc python-devel
If you are seeing this message on Microsoft Windows please install
PyMongo using pip. Modern versions of pip will install PyMongo
from binary wheels available on pypi. If you must install from
source read the documentation here:
https://pymongo.readthedocs.io/en/stable/installation.html#installing-from-source-on-windows
If you are seeing this message on macOS / OSX please install PyMongo
using pip. Modern versions of pip will install PyMongo from binary
wheels available on pypi. If wheels are not available for your version
of macOS / OSX, or you must install from source read the documentation
here:
https://pymongo.readthedocs.io/en/stable/installation.html#osx
********************************************************************
"""
def run(self):
try:
build_ext.run(self)
except Exception:
if os.environ.get("PYMONGO_C_EXT_MUST_BUILD"):
raise
e = sys.exc_info()[1]
sys.stdout.write("%s\n" % str(e))
warnings.warn(
self.warning_message
% (
"Extension modules",
"There was an issue with your platform configuration - see above.",
),
stacklevel=2,
)
def build_extension(self, ext):
# "ProgramFiles(x86)" is not a valid environment variable in Cygwin but is needed for
# the MSVCCompiler in distutils.
if os.name == "nt":
if "ProgramFiles" in os.environ and "ProgramFiles(x86)" not in os.environ:
os.environ["ProgramFiles(x86)"] = os.environ["ProgramFiles"] + " (x86)"
name = ext.name
try:
build_ext.build_extension(self, ext)
except Exception:
if os.environ.get("PYMONGO_C_EXT_MUST_BUILD"):
raise
e = sys.exc_info()[1]
sys.stdout.write("%s\n" % str(e))
warnings.warn(
self.warning_message
% (
"The %s extension module" % (name,), # noqa: UP031
"The output above this warning shows how the compilation failed.",
),
stacklevel=2,
)
ext_modules = [
Extension(
"bson._cbson",
include_dirs=["bson"],
sources=["bson/_cbsonmodule.c", "bson/time64.c", "bson/buffer.c"],
),
Extension(
"pymongo._cmessage",
include_dirs=["bson"],
sources=[
"pymongo/_cmessagemodule.c",
"bson/_cbsonmodule.c",
"bson/time64.c",
"bson/buffer.c",
],
),
]
if "--no_ext" in sys.argv or os.environ.get("NO_EXT"):
try:
sys.argv.remove("--no_ext")
except ValueError:
pass
ext_modules = []
elif (
sys.platform.startswith("java")
or sys.platform == "cli"
or sys.implementation.name in ("pypy", "graalpy")
):
sys.stdout.write(
"""
*****************************************************\n
The optional C extensions are currently not supported\n
by this python implementation.\n
*****************************************************\n
"""
)
ext_modules = []
setup(
cmdclass={"build_ext": custom_build_ext},
ext_modules=ext_modules,
packages=["bson", "pymongo", "gridfs"],
) # type:ignore

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

View File

@ -1,5 +1,5 @@
/* /*
* Copyright 2009-present MongoDB, Inc. * Copyright 2009-2015 MongoDB, Inc.
* *
* Licensed under the Apache License, Version 2.0 (the "License"); * Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License. * you may not use this file except in compliance with the License.
@ -14,67 +14,44 @@
* limitations under the License. * limitations under the License.
*/ */
#include "bson-endian.h"
#ifndef _CBSONMODULE_H #ifndef _CBSONMODULE_H
#define _CBSONMODULE_H #define _CBSONMODULE_H
#if defined(WIN32) || defined(_MSC_VER) /* Py_ssize_t was new in python 2.5. See conversion
* guidlines in http://www.python.org/dev/peps/pep-0353
* */
#if PY_VERSION_HEX < 0x02050000 && !defined(PY_SSIZE_T_MIN)
typedef int Py_ssize_t;
#define PY_SSIZE_T_MAX INT_MAX
#define PY_SSIZE_T_MIN INT_MIN
#endif
#ifdef _MSC_VER
/* /*
* This macro is basically an implementation of asprintf for win32 * This macro is basically an implementation of asprintf for win32
* We print to the provided buffer to get the string value as an int. * We print to the provided buffer to get the string value as an int.
* USE LL2STR. This is kept only to test LL2STR.
*/ */
#if defined(_MSC_VER) && (_MSC_VER >= 1400) #if _MSC_VER >= 1400
#define INT2STRING(buffer, i) \ #define INT2STRING(buffer, i) \
_snprintf_s((buffer), \ _snprintf_s((buffer), \
_scprintf("%lld", (i)) + 1, \ _scprintf("%d", (i)) + 1, \
_scprintf("%lld", (i)) + 1, \ _scprintf("%d", (i)) + 1, \
"%lld", \ "%d", \
(i)) (i))
#define STRCAT(dest, n, src) strcat_s((dest), (n), (src)) #define STRCAT(dest, n, src) strcat_s((dest), (n), (src))
#else #else
#define INT2STRING(buffer, i) \ #define INT2STRING(buffer, i) \
_snprintf((buffer), \ _snprintf((buffer), \
_scprintf("%lld", (i)) + 1, \ _scprintf("%d", (i)) + 1, \
"%lld", \ "%d", \
(i)) (i))
#define STRCAT(dest, n, src) strcat((dest), (src)) #define STRCAT(dest, n, src) strcat((dest), (src))
#endif #endif
#else #else
#define INT2STRING(buffer, i) snprintf((buffer), sizeof((buffer)), "%lld", (i)) #define INT2STRING(buffer, i) snprintf((buffer), sizeof((buffer)), "%d", (i))
#define STRCAT(dest, n, src) strcat((dest), (src)) #define STRCAT(dest, n, src) strcat((dest), (src))
#endif #endif
/* Just enough space in char array to hold LLONG_MIN and null terminator */
#define BUF_SIZE 21
/* Converts integer to its string representation in decimal notation. */
extern int cbson_long_long_to_str(long long int num, char* str, size_t size);
#define LL2STR(buffer, i) cbson_long_long_to_str((i), (buffer), sizeof(buffer))
typedef struct type_registry_t {
PyObject* encoder_map;
PyObject* decoder_map;
PyObject* fallback_encoder;
PyObject* registry_obj;
unsigned char is_encoder_empty;
unsigned char is_decoder_empty;
unsigned char has_fallback_encoder;
} type_registry_t;
typedef struct codec_options_t {
PyObject* document_class;
unsigned char tz_aware;
unsigned char uuid_rep;
char* unicode_decode_error_handler;
PyObject* tzinfo;
type_registry_t type_registry;
unsigned char datetime_conversion;
PyObject* options_obj;
unsigned char is_raw_bson;
unsigned char is_dict_class;
} codec_options_t;
/* C API functions */ /* C API functions */
#define _cbson_buffer_write_bytes_INDEX 0 #define _cbson_buffer_write_bytes_INDEX 0
#define _cbson_buffer_write_bytes_RETURN int #define _cbson_buffer_write_bytes_RETURN int
@ -82,46 +59,18 @@ typedef struct codec_options_t {
#define _cbson_write_dict_INDEX 1 #define _cbson_write_dict_INDEX 1
#define _cbson_write_dict_RETURN int #define _cbson_write_dict_RETURN int
#define _cbson_write_dict_PROTO (PyObject* self, buffer_t buffer, PyObject* dict, unsigned char check_keys, const codec_options_t* options, unsigned char top_level) #define _cbson_write_dict_PROTO (PyObject* self, buffer_t buffer, PyObject* dict, unsigned char check_keys, unsigned char uuid_subtype, unsigned char top_level)
#define _cbson_write_pair_INDEX 2 #define _cbson_write_pair_INDEX 2
#define _cbson_write_pair_RETURN int #define _cbson_write_pair_RETURN int
#define _cbson_write_pair_PROTO (PyObject* self, buffer_t buffer, const char* name, int name_length, PyObject* value, unsigned char check_keys, const codec_options_t* options, unsigned char allow_id) #define _cbson_write_pair_PROTO (PyObject* self, buffer_t buffer, const char* name, int name_length, PyObject* value, unsigned char check_keys, unsigned char uuid_subtype, unsigned char allow_id)
#define _cbson_decode_and_write_pair_INDEX 3 #define _cbson_decode_and_write_pair_INDEX 3
#define _cbson_decode_and_write_pair_RETURN int #define _cbson_decode_and_write_pair_RETURN int
#define _cbson_decode_and_write_pair_PROTO (PyObject* self, buffer_t buffer, PyObject* key, PyObject* value, unsigned char check_keys, const codec_options_t* options, unsigned char top_level) #define _cbson_decode_and_write_pair_PROTO (PyObject* self, buffer_t buffer, PyObject* key, PyObject* value, unsigned char check_keys, unsigned char uuid_subtype, unsigned char top_level)
#define _cbson_convert_codec_options_INDEX 4
#define _cbson_convert_codec_options_RETURN int
#define _cbson_convert_codec_options_PROTO (PyObject* self, PyObject* options_obj, codec_options_t* options)
#define _cbson_destroy_codec_options_INDEX 5
#define _cbson_destroy_codec_options_RETURN void
#define _cbson_destroy_codec_options_PROTO (codec_options_t* options)
#define _cbson_buffer_write_double_INDEX 6
#define _cbson_buffer_write_double_RETURN int
#define _cbson_buffer_write_double_PROTO (buffer_t buffer, double data)
#define _cbson_buffer_write_int32_INDEX 7
#define _cbson_buffer_write_int32_RETURN int
#define _cbson_buffer_write_int32_PROTO (buffer_t buffer, int32_t data)
#define _cbson_buffer_write_int64_INDEX 8
#define _cbson_buffer_write_int64_RETURN int
#define _cbson_buffer_write_int64_PROTO (buffer_t buffer, int64_t data)
#define _cbson_buffer_write_int32_at_position_INDEX 9
#define _cbson_buffer_write_int32_at_position_RETURN void
#define _cbson_buffer_write_int32_at_position_PROTO (buffer_t buffer, int position, int32_t data)
#define _cbson_downcast_and_check_INDEX 10
#define _cbson_downcast_and_check_RETURN int
#define _cbson_downcast_and_check_PROTO (Py_ssize_t size, uint8_t extra)
/* Total number of C API pointers */ /* Total number of C API pointers */
#define _cbson_API_POINTER_COUNT 11 #define _cbson_API_POINTER_COUNT 4
#ifdef _CBSON_MODULE #ifdef _CBSON_MODULE
/* This section is used when compiling _cbsonmodule */ /* This section is used when compiling _cbsonmodule */
@ -134,20 +83,6 @@ static _cbson_write_pair_RETURN write_pair _cbson_write_pair_PROTO;
static _cbson_decode_and_write_pair_RETURN decode_and_write_pair _cbson_decode_and_write_pair_PROTO; static _cbson_decode_and_write_pair_RETURN decode_and_write_pair _cbson_decode_and_write_pair_PROTO;
static _cbson_convert_codec_options_RETURN convert_codec_options _cbson_convert_codec_options_PROTO;
static _cbson_destroy_codec_options_RETURN destroy_codec_options _cbson_destroy_codec_options_PROTO;
static _cbson_buffer_write_double_RETURN buffer_write_double _cbson_buffer_write_double_PROTO;
static _cbson_buffer_write_int32_RETURN buffer_write_int32 _cbson_buffer_write_int32_PROTO;
static _cbson_buffer_write_int64_RETURN buffer_write_int64 _cbson_buffer_write_int64_PROTO;
static _cbson_buffer_write_int32_at_position_RETURN buffer_write_int32_at_position _cbson_buffer_write_int32_at_position_PROTO;
static _cbson_downcast_and_check_RETURN _downcast_and_check _cbson_downcast_and_check_PROTO;
#else #else
/* This section is used in modules that use _cbsonmodule's API */ /* This section is used in modules that use _cbsonmodule's API */
@ -161,20 +96,6 @@ static void **_cbson_API;
#define decode_and_write_pair (*(_cbson_decode_and_write_pair_RETURN (*)_cbson_decode_and_write_pair_PROTO) _cbson_API[_cbson_decode_and_write_pair_INDEX]) #define decode_and_write_pair (*(_cbson_decode_and_write_pair_RETURN (*)_cbson_decode_and_write_pair_PROTO) _cbson_API[_cbson_decode_and_write_pair_INDEX])
#define convert_codec_options (*(_cbson_convert_codec_options_RETURN (*)_cbson_convert_codec_options_PROTO) _cbson_API[_cbson_convert_codec_options_INDEX])
#define destroy_codec_options (*(_cbson_destroy_codec_options_RETURN (*)_cbson_destroy_codec_options_PROTO) _cbson_API[_cbson_destroy_codec_options_INDEX])
#define buffer_write_double (*(_cbson_buffer_write_double_RETURN (*)_cbson_buffer_write_double_PROTO) _cbson_API[_cbson_buffer_write_double_INDEX])
#define buffer_write_int32 (*(_cbson_buffer_write_int32_RETURN (*)_cbson_buffer_write_int32_PROTO) _cbson_API[_cbson_buffer_write_int32_INDEX])
#define buffer_write_int64 (*(_cbson_buffer_write_int64_RETURN (*)_cbson_buffer_write_int64_PROTO) _cbson_API[_cbson_buffer_write_int64_INDEX])
#define buffer_write_int32_at_position (*(_cbson_buffer_write_int32_at_position_RETURN (*)_cbson_buffer_write_int32_at_position_PROTO) _cbson_API[_cbson_buffer_write_int32_at_position_INDEX])
#define _downcast_and_check (*(_cbson_downcast_and_check_RETURN (*)_cbson_downcast_and_check_PROTO) _cbson_API[_cbson_downcast_and_check_INDEX])
#define _cbson_IMPORT _cbson_API = (void **)PyCapsule_Import("_cbson._C_API", 0) #define _cbson_IMPORT _cbson_API = (void **)PyCapsule_Import("_cbson._C_API", 0)
#endif #endif

View File

@ -1,43 +0,0 @@
# Copyright 2021-present MongoDB, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Setstate and getstate functions for objects with __slots__, allowing
compatibility with default pickling protocol
"""
from __future__ import annotations
from typing import Any, Mapping
def _setstate_slots(self: Any, state: Any) -> None:
for slot, value in state.items():
setattr(self, slot, value)
def _mangle_name(name: str, prefix: str) -> str:
if name.startswith("__"):
prefix = "_" + prefix
else:
prefix = ""
return prefix + name
def _getstate_slots(self: Any) -> Mapping[Any, Any]:
prefix = self.__class__.__name__
ret = {}
for name in self.__slots__:
mangled_name = _mangle_name(name, prefix)
if hasattr(self, mangled_name):
ret[mangled_name] = getattr(self, mangled_name)
return ret

View File

@ -1,4 +1,4 @@
# Copyright 2009-present MongoDB, Inc. # Copyright 2009-2015 MongoDB, Inc.
# #
# Licensed under the Apache License, Version 2.0 (the "License"); # Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License. # you may not use this file except in compliance with the License.
@ -11,13 +11,14 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
from __future__ import annotations
import struct try:
import warnings from uuid import UUID
from enum import Enum except ImportError:
from typing import TYPE_CHECKING, Any, Optional, Sequence, Tuple, Type, Union, overload # Python2.4 doesn't have a uuid module.
from uuid import UUID pass
from bson.py3compat import PY3, binary_type
"""Tools for representing BSON binary data. """Tools for representing BSON binary data.
""" """
@ -26,10 +27,14 @@ BINARY_SUBTYPE = 0
"""BSON binary subtype for binary data. """BSON binary subtype for binary data.
This is the default subtype for binary data. This is the default subtype for binary data.
.. versionadded:: 1.5
""" """
FUNCTION_SUBTYPE = 1 FUNCTION_SUBTYPE = 1
"""BSON binary subtype for functions. """BSON binary subtype for functions.
.. versionadded:: 1.5
""" """
OLD_BINARY_SUBTYPE = 2 OLD_BINARY_SUBTYPE = 2
@ -37,16 +42,15 @@ OLD_BINARY_SUBTYPE = 2
This is the old default subtype, the current This is the old default subtype, the current
default is :data:`BINARY_SUBTYPE`. default is :data:`BINARY_SUBTYPE`.
.. versionadded:: 1.7
""" """
OLD_UUID_SUBTYPE = 3 OLD_UUID_SUBTYPE = 3
"""Old BSON binary subtype for a UUID. """Old BSON binary subtype for a UUID.
:class:`uuid.UUID` instances will automatically be encoded :class:`uuid.UUID` instances will automatically be encoded
by :mod:`bson` using this subtype when using by :mod:`bson` using this subtype.
:data:`UuidRepresentation.PYTHON_LEGACY`,
:data:`UuidRepresentation.JAVA_LEGACY`, or
:data:`UuidRepresentation.CSHARP_LEGACY`.
.. versionadded:: 2.1 .. versionadded:: 2.1
""" """
@ -54,612 +58,201 @@ by :mod:`bson` using this subtype when using
UUID_SUBTYPE = 4 UUID_SUBTYPE = 4
"""BSON binary subtype for a UUID. """BSON binary subtype for a UUID.
This is the standard BSON binary subtype for UUIDs. This is the new BSON binary subtype for UUIDs. The
:class:`uuid.UUID` instances will automatically be encoded current default is :data:`OLD_UUID_SUBTYPE` but will
by :mod:`bson` using this subtype when using change to this in a future release.
:data:`UuidRepresentation.STANDARD`.
.. versionchanged:: 2.1
Changed to subtype 4.
.. versionadded:: 1.5
""" """
STANDARD = UUID_SUBTYPE
"""The standard UUID representation.
if TYPE_CHECKING: :class:`uuid.UUID` instances will automatically be encoded to
from array import array as _array and decoded from BSON binary, using RFC-4122 byte order with
from mmap import mmap as _mmap binary subtype :data:`UUID_SUBTYPE`.
import numpy as np .. versionadded:: 2.9
import numpy.typing as npt
class UuidRepresentation:
UNSPECIFIED = 0
"""An unspecified UUID representation.
When configured, :class:`uuid.UUID` instances will **not** be
automatically encoded to or decoded from :class:`~bson.binary.Binary`.
When encoding a :class:`uuid.UUID` instance, an error will be raised.
To encode a :class:`uuid.UUID` instance with this configuration, it must
be wrapped in the :class:`~bson.binary.Binary` class by the application
code. When decoding a BSON binary field with a UUID subtype, a
:class:`~bson.binary.Binary` instance will be returned instead of a
:class:`uuid.UUID` instance.
See `unspecified representation details <https://www.mongodb.com/docs/languages/python/pymongo-driver/current/data-formats/uuid/#unspecified>`_ for details.
.. versionadded:: 3.11
"""
STANDARD = UUID_SUBTYPE
"""The standard UUID representation.
:class:`uuid.UUID` instances will automatically be encoded to
and decoded from BSON binary, using RFC-4122 byte order with
binary subtype :data:`UUID_SUBTYPE`.
See `standard representation details <https://www.mongodb.com/docs/languages/python/pymongo-driver/current/data-formats/uuid/#standard>`_ for details.
.. versionadded:: 3.11
"""
PYTHON_LEGACY = OLD_UUID_SUBTYPE
"""The Python legacy UUID representation.
:class:`uuid.UUID` instances will automatically be encoded to
and decoded from BSON binary, using RFC-4122 byte order with
binary subtype :data:`OLD_UUID_SUBTYPE`.
See `python legacy representation details <https://www.mongodb.com/docs/languages/python/pymongo-driver/current/data-formats/uuid/#python_legacy>`_ for details.
.. versionadded:: 3.11
"""
JAVA_LEGACY = 5
"""The Java legacy UUID representation.
:class:`uuid.UUID` instances will automatically be encoded to
and decoded from BSON binary subtype :data:`OLD_UUID_SUBTYPE`,
using the Java driver's legacy byte order.
See `Java Legacy UUID <https://www.mongodb.com/docs/languages/python/pymongo-driver/current/data-formats/uuid/#java_legacy>`_ for details.
.. versionadded:: 3.11
"""
CSHARP_LEGACY = 6
"""The C#/.net legacy UUID representation.
:class:`uuid.UUID` instances will automatically be encoded to
and decoded from BSON binary subtype :data:`OLD_UUID_SUBTYPE`,
using the C# driver's legacy byte order.
See `C# Legacy UUID <https://www.mongodb.com/docs/languages/python/pymongo-driver/current/data-formats/uuid/#csharp_legacy>`_ for details.
.. versionadded:: 3.11
"""
STANDARD = UuidRepresentation.STANDARD
"""An alias for :data:`UuidRepresentation.STANDARD`.
.. versionadded:: 3.0
""" """
PYTHON_LEGACY = UuidRepresentation.PYTHON_LEGACY PYTHON_LEGACY = OLD_UUID_SUBTYPE
"""An alias for :data:`UuidRepresentation.PYTHON_LEGACY`. """The Python legacy UUID representation.
.. versionadded:: 3.0 :class:`uuid.UUID` instances will automatically be encoded to
and decoded from BSON binary, using RFC-4122 byte order with
binary subtype :data:`OLD_UUID_SUBTYPE`.
.. versionadded:: 2.9
""" """
JAVA_LEGACY = UuidRepresentation.JAVA_LEGACY JAVA_LEGACY = 5
"""An alias for :data:`UuidRepresentation.JAVA_LEGACY`. """The Java legacy UUID representation.
:class:`uuid.UUID` instances will automatically be encoded to
and decoded from BSON binary, using the Java driver's legacy
byte order with binary subtype :data:`OLD_UUID_SUBTYPE`.
.. versionchanged:: 3.6
BSON binary subtype 4 is decoded using RFC-4122 byte order.
.. versionadded:: 2.3 .. versionadded:: 2.3
""" """
CSHARP_LEGACY = UuidRepresentation.CSHARP_LEGACY CSHARP_LEGACY = 6
"""An alias for :data:`UuidRepresentation.CSHARP_LEGACY`. """The C#/.net legacy UUID representation.
:class:`uuid.UUID` instances will automatically be encoded to
and decoded from BSON binary, using the C# driver's legacy
byte order and binary subtype :data:`OLD_UUID_SUBTYPE`.
.. versionchanged:: 3.6
BSON binary subtype 4 is decoded using RFC-4122 byte order.
.. versionadded:: 2.3 .. versionadded:: 2.3
""" """
ALL_UUID_SUBTYPES = (OLD_UUID_SUBTYPE, UUID_SUBTYPE) ALL_UUID_SUBTYPES = (OLD_UUID_SUBTYPE, UUID_SUBTYPE, JAVA_LEGACY, CSHARP_LEGACY)
ALL_UUID_REPRESENTATIONS = ( ALL_UUID_REPRESENTATIONS = (STANDARD, PYTHON_LEGACY, JAVA_LEGACY, CSHARP_LEGACY)
UuidRepresentation.UNSPECIFIED,
UuidRepresentation.STANDARD,
UuidRepresentation.PYTHON_LEGACY,
UuidRepresentation.JAVA_LEGACY,
UuidRepresentation.CSHARP_LEGACY,
)
UUID_REPRESENTATION_NAMES = { UUID_REPRESENTATION_NAMES = {
UuidRepresentation.UNSPECIFIED: "UuidRepresentation.UNSPECIFIED", PYTHON_LEGACY: 'PYTHON_LEGACY',
UuidRepresentation.STANDARD: "UuidRepresentation.STANDARD", STANDARD: 'STANDARD',
UuidRepresentation.PYTHON_LEGACY: "UuidRepresentation.PYTHON_LEGACY", JAVA_LEGACY: 'JAVA_LEGACY',
UuidRepresentation.JAVA_LEGACY: "UuidRepresentation.JAVA_LEGACY", CSHARP_LEGACY: 'CSHARP_LEGACY'}
UuidRepresentation.CSHARP_LEGACY: "UuidRepresentation.CSHARP_LEGACY",
}
MD5_SUBTYPE = 5 MD5_SUBTYPE = 5
"""BSON binary subtype for an MD5 hash. """BSON binary subtype for an MD5 hash.
.. versionadded:: 1.5
""" """
COLUMN_SUBTYPE = 7
"""BSON binary subtype for columns.
.. versionadded:: 4.0
"""
SENSITIVE_SUBTYPE = 8
"""BSON binary subtype for sensitive data.
.. versionadded:: 4.5
"""
VECTOR_SUBTYPE = 9
"""BSON binary subtype for densely packed vector data.
.. versionadded:: 4.10
"""
USER_DEFINED_SUBTYPE = 128 USER_DEFINED_SUBTYPE = 128
"""BSON binary subtype for any user defined structure. """BSON binary subtype for any user defined structure.
.. versionadded:: 1.5
""" """
class BinaryVectorDtype(Enum): class Binary(binary_type):
"""Datatypes of vector subtype.
:param FLOAT32: (0x27) Pack list of :class:`float` as float32
:param INT8: (0x03) Pack list of :class:`int` in [-128, 127] as signed int8
:param PACKED_BIT: (0x10) Pack list of :class:`int` in [0, 255] as unsigned uint8
The `PACKED_BIT` value represents a special case where vector values themselves
can only be of two values (0 or 1) but these are packed together into groups of 8,
a byte. In Python, these are displayed as ints in range [0, 255]
Each value is of type bytes with a length of one.
.. versionadded:: 4.10
"""
INT8 = b"\x03"
FLOAT32 = b"\x27"
PACKED_BIT = b"\x10"
class BinaryVector:
"""Vector of numbers along with metadata for binary interoperability.
.. versionadded:: 4.10
"""
__slots__ = ("data", "dtype", "padding")
def __init__(
self,
data: Union[Sequence[float | int], npt.NDArray[np.number]],
dtype: BinaryVectorDtype,
padding: int = 0,
):
"""
:param data: Sequence of numbers representing the mathematical vector.
:param dtype: The data type stored in binary
:param padding: The number of bits in the final byte that are to be ignored
when a vector element's size is less than a byte
and the length of the vector is not a multiple of 8.
(Padding is equivalent to a negative value of `count` in
`numpy.unpackbits <https://numpy.org/doc/stable/reference/generated/numpy.unpackbits.html>`_)
"""
self.data = data
self.dtype = dtype
self.padding = padding
def __repr__(self) -> str:
return f"BinaryVector(dtype={self.dtype}, padding={self.padding}, data={self.data})"
def __eq__(self, other: Any) -> bool:
if not isinstance(other, BinaryVector):
return False
return (
self.dtype == other.dtype and self.padding == other.padding and self.data == other.data
)
def __len__(self) -> int:
return len(self.data)
class Binary(bytes):
"""Representation of BSON binary data. """Representation of BSON binary data.
We want to represent Python strings as the BSON string type. This is necessary because we want to represent Python strings as
We need to wrap binary data so that we can tell the BSON string type. We need to wrap binary data so we can tell
the difference between what should be considered binary data and the difference between what should be considered binary data and
what should be considered a string when we encode to BSON. what should be considered a string when we encode to BSON.
Subtype 9 provides a space-efficient representation of 1-dimensional vector data. Raises TypeError if `data` is not an instance of :class:`str`
Its data is prepended with two bytes of metadata. (:class:`bytes` in python 3) or `subtype` is not an instance of
The first (dtype) describes its data type, such as float32 or int8. :class:`int`. Raises ValueError if `subtype` is not in [0, 256).
The second (padding) prescribes the number of bits to ignore in the final byte.
This is relevant when the element size of the dtype is not a multiple of 8.
Raises TypeError if `subtype` is not an instance of :class:`int`.
Raises ValueError if `subtype` is not in [0, 256).
.. note:: .. note::
Instances of Binary with subtype 0 will be decoded directly to :class:`bytes`. In python 3 instances of Binary with subtype 0 will be decoded
directly to :class:`bytes`.
:param data: the binary data to represent. Can be any bytes-like type :Parameters:
that implements the buffer protocol. - `data`: the binary data to represent
:param subtype: the `binary subtype - `subtype` (optional): the `binary subtype
<https://bsonspec.org/spec.html>`_ <http://bsonspec.org/#/specification>`_
to use to use
.. versionchanged:: 3.9
Support any bytes-like type that implements the buffer protocol.
.. versionchanged:: 4.10
Addition of vector subtype.
""" """
_type_marker = 5 _type_marker = 5
__subtype: int
def __new__( def __new__(cls, data, subtype=BINARY_SUBTYPE):
cls: Type[Binary], if not isinstance(data, binary_type):
data: Union[memoryview, bytes, bytearray, _mmap, _array[Any]], raise TypeError("data must be an "
subtype: int = BINARY_SUBTYPE, "instance of %s" % (binary_type.__name__,))
) -> Binary:
if not isinstance(subtype, int): if not isinstance(subtype, int):
raise TypeError(f"subtype must be an instance of int, not {type(subtype)}") raise TypeError("subtype must be an instance of int")
if subtype >= 256 or subtype < 0: if subtype >= 256 or subtype < 0:
raise ValueError("subtype must be contained in [0, 256)") raise ValueError("subtype must be contained in [0, 256)")
# Support any type that implements the buffer protocol. self = binary_type.__new__(cls, data)
self = bytes.__new__(cls, memoryview(data).tobytes())
self.__subtype = subtype self.__subtype = subtype
return self return self
@classmethod
def from_uuid(
cls: Type[Binary], uuid: UUID, uuid_representation: int = UuidRepresentation.STANDARD
) -> Binary:
"""Create a BSON Binary object from a Python UUID.
Creates a :class:`~bson.binary.Binary` object from a
:class:`uuid.UUID` instance. Assumes that the native
:class:`uuid.UUID` instance uses the byte-order implied by the
provided ``uuid_representation``.
Raises :exc:`TypeError` if `uuid` is not an instance of
:class:`~uuid.UUID`.
:param uuid: A :class:`uuid.UUID` instance.
:param uuid_representation: A member of
:class:`~bson.binary.UuidRepresentation`. Default:
:const:`~bson.binary.UuidRepresentation.STANDARD`.
See `UUID representations <https://www.mongodb.com/docs/languages/python/pymongo-driver/current/data-formats/uuid/#universally-unique-ids--uuids->`_ for details.
.. versionadded:: 3.11
"""
if not isinstance(uuid, UUID):
raise TypeError(f"uuid must be an instance of uuid.UUID, not {type(uuid)}")
if uuid_representation not in ALL_UUID_REPRESENTATIONS:
raise ValueError(
"uuid_representation must be a value from bson.binary.UuidRepresentation"
)
if uuid_representation == UuidRepresentation.UNSPECIFIED:
raise ValueError(
"cannot encode native uuid.UUID with "
"UuidRepresentation.UNSPECIFIED. UUIDs can be manually "
"converted to bson.Binary instances using "
"bson.Binary.from_uuid() or a different UuidRepresentation "
"can be configured. See the documentation for "
"UuidRepresentation for more information."
)
subtype = OLD_UUID_SUBTYPE
if uuid_representation == UuidRepresentation.PYTHON_LEGACY:
payload = uuid.bytes
elif uuid_representation == UuidRepresentation.JAVA_LEGACY:
from_uuid = uuid.bytes
payload = from_uuid[0:8][::-1] + from_uuid[8:16][::-1]
elif uuid_representation == UuidRepresentation.CSHARP_LEGACY:
payload = uuid.bytes_le
else:
# uuid_representation == UuidRepresentation.STANDARD
subtype = UUID_SUBTYPE
payload = uuid.bytes
return cls(payload, subtype)
def as_uuid(self, uuid_representation: int = UuidRepresentation.STANDARD) -> UUID:
"""Create a Python UUID from this BSON Binary object.
Decodes this binary object as a native :class:`uuid.UUID` instance
with the provided ``uuid_representation``.
Raises :exc:`ValueError` if this :class:`~bson.binary.Binary` instance
does not contain a UUID.
:param uuid_representation: A member of
:class:`~bson.binary.UuidRepresentation`. Default:
:const:`~bson.binary.UuidRepresentation.STANDARD`.
See `UUID representations <https://www.mongodb.com/docs/languages/python/pymongo-driver/current/data-formats/uuid/#universally-unique-ids--uuids->`_ for details.
.. versionadded:: 3.11
"""
if self.subtype not in ALL_UUID_SUBTYPES:
raise ValueError(f"cannot decode subtype {self.subtype} as a uuid")
if uuid_representation not in ALL_UUID_REPRESENTATIONS:
raise ValueError(
"uuid_representation must be a value from bson.binary.UuidRepresentation"
)
if uuid_representation == UuidRepresentation.UNSPECIFIED:
raise ValueError("uuid_representation cannot be UNSPECIFIED")
elif uuid_representation == UuidRepresentation.PYTHON_LEGACY:
if self.subtype == OLD_UUID_SUBTYPE:
return UUID(bytes=self)
elif uuid_representation == UuidRepresentation.JAVA_LEGACY:
if self.subtype == OLD_UUID_SUBTYPE:
return UUID(bytes=self[0:8][::-1] + self[8:16][::-1])
elif uuid_representation == UuidRepresentation.CSHARP_LEGACY:
if self.subtype == OLD_UUID_SUBTYPE:
return UUID(bytes_le=self)
else:
# uuid_representation == UuidRepresentation.STANDARD
if self.subtype == UUID_SUBTYPE:
return UUID(bytes=self)
raise ValueError(
f"cannot decode subtype {self.subtype} to {UUID_REPRESENTATION_NAMES[uuid_representation]}"
)
@classmethod
@overload
def from_vector(cls: Type[Binary], vector: BinaryVector) -> Binary:
...
@classmethod
@overload
def from_vector(
cls: Type[Binary],
vector: Union[list[int], list[float]],
dtype: BinaryVectorDtype,
padding: int = 0,
) -> Binary:
...
@classmethod
@overload
def from_vector(
cls: Type[Binary],
vector: npt.NDArray[np.number],
dtype: BinaryVectorDtype,
padding: int = 0,
) -> Binary:
...
@classmethod
def from_vector(
cls: Type[Binary],
vector: Union[BinaryVector, list[int], list[float], npt.NDArray[np.number]],
dtype: Optional[BinaryVectorDtype] = None,
padding: Optional[int] = None,
) -> Binary:
"""Create a BSON :class:`~bson.binary.Binary` of Vector subtype.
To interpret the representation of the numbers, a data type must be included.
See :class:`~bson.binary.BinaryVectorDtype` for available types and descriptions.
The dtype and padding are prepended to the binary data's value.
:param vector: Either a List of values, or a :class:`~bson.binary.BinaryVector` dataclass.
:param dtype: Data type of the values
:param padding: For fractional bytes, number of bits to ignore at end of vector.
:return: Binary packed data identified by dtype and padding.
.. versionchanged:: 4.14
When padding is non-zero, ignored bits should be zero. Raise exception on encoding, warn on decoding.
.. versionadded:: 4.10
"""
if isinstance(vector, BinaryVector):
if dtype or padding:
raise ValueError(
"The first argument, vector, has type BinaryVector. "
"dtype or padding cannot be separately defined, but were."
)
dtype = vector.dtype
padding = vector.padding
vector = vector.data # type: ignore
padding = 0 if padding is None else padding
if not isinstance(dtype, BinaryVectorDtype):
raise TypeError(
"dtype must be a bson.BinaryVectorDtype of BinaryVectorDType.INT8, PACKED_BIT, FLOAT32"
)
metadata = struct.pack("<sB", dtype.value, padding)
if isinstance(vector, list):
if dtype == BinaryVectorDtype.INT8: # pack ints in [-128, 127] as signed int8
format_str = "b"
if padding:
raise ValueError(f"padding does not apply to {dtype=}")
elif dtype == BinaryVectorDtype.PACKED_BIT: # pack ints in [0, 255] as unsigned uint8
format_str = "B"
if 0 <= padding > 7:
raise ValueError(f"{padding=}. It must be in [0,1, ..7].")
if padding and not vector:
raise ValueError("Empty vector with non-zero padding.")
elif dtype == BinaryVectorDtype.FLOAT32: # pack floats as float32
format_str = "f"
if padding:
raise ValueError(f"padding does not apply to {dtype=}")
else:
raise NotImplementedError("%s not yet supported" % dtype)
data = struct.pack(f"<{len(vector)}{format_str}", *vector)
else: # vector is numpy array or incorrect type.
try:
import numpy as np
except ImportError as exc:
raise ImportError(
"Failed to create binary from vector. Check type. If numpy array, numpy must be installed."
) from exc
if not isinstance(vector, np.ndarray):
raise TypeError(
"Could not create Binary. Vector must be a BinaryVector, list[int], list[float] or numpy ndarray."
)
if vector.ndim != 1:
raise ValueError(
"from_numpy_vector only supports 1D arrays as it creates a single vector."
)
if dtype == BinaryVectorDtype.FLOAT32:
vector = vector.astype(np.dtype("float32"), copy=False)
elif dtype == BinaryVectorDtype.INT8:
if vector.min() >= -128 and vector.max() <= 127:
vector = vector.astype(np.dtype("int8"), copy=False)
else:
raise ValueError("Values found outside INT8 range.")
elif dtype == BinaryVectorDtype.PACKED_BIT:
if vector.min() >= 0 and vector.max() <= 127:
vector = vector.astype(np.dtype("uint8"), copy=False)
else:
raise ValueError("Values found outside UINT8 range.")
else:
raise NotImplementedError("%s not yet supported" % dtype)
data = vector.tobytes()
if padding and len(vector) and not (data[-1] & ((1 << padding) - 1)) == 0:
raise ValueError(
"Vector has a padding P, but bits in the final byte lower than P are non-zero. They must be zero."
)
return cls(metadata + data, subtype=VECTOR_SUBTYPE)
def as_vector(self, return_numpy: bool = False) -> BinaryVector:
"""From the Binary, create a list or 1-d numpy array of numbers, along with dtype and padding.
:param return_numpy: If True, BinaryVector.data will be a one-dimensional numpy array. By default, it is a list.
:return: BinaryVector
.. versionadded:: 4.10
"""
if self.subtype != VECTOR_SUBTYPE:
raise ValueError(f"Cannot decode subtype {self.subtype} as a vector")
dtype, padding = struct.unpack_from("<sB", self)
dtype = BinaryVectorDtype(dtype)
offset = 2
n_bytes = len(self) - offset
if padding and dtype != BinaryVectorDtype.PACKED_BIT:
raise ValueError(
f"Corrupt data. Padding ({padding}) must be 0 for all but PACKED_BIT dtypes. ({dtype=})"
)
if not return_numpy:
if dtype == BinaryVectorDtype.INT8:
dtype_format = "b"
format_string = f"<{n_bytes}{dtype_format}"
vector = list(struct.unpack_from(format_string, self, offset))
return BinaryVector(vector, dtype, padding)
elif dtype == BinaryVectorDtype.FLOAT32:
n_values = n_bytes // 4
if n_bytes % 4:
raise ValueError(
"Corrupt data. N bytes for a float32 vector must be a multiple of 4."
)
dtype_format = "f"
format_string = f"<{n_values}{dtype_format}"
vector = list(struct.unpack_from(format_string, self, offset))
return BinaryVector(vector, dtype, padding)
elif dtype == BinaryVectorDtype.PACKED_BIT:
# data packed as uint8
if padding and not n_bytes:
raise ValueError("Corrupt data. Vector has a padding P, but no data.")
if padding > 7 or padding < 0:
raise ValueError(f"Corrupt data. Padding ({padding}) must be between 0 and 7.")
dtype_format = "B"
format_string = f"<{n_bytes}{dtype_format}"
unpacked_uint8s = list(struct.unpack_from(format_string, self, offset))
if padding and n_bytes and unpacked_uint8s[-1] & (1 << padding) - 1 != 0:
warnings.warn(
"Vector has a padding P, but bits in the final byte lower than P are non-zero. For pymongo>=5.0, they must be zero.",
DeprecationWarning,
stacklevel=2,
)
return BinaryVector(unpacked_uint8s, dtype, padding)
else:
raise NotImplementedError("Binary Vector dtype %s not yet supported" % dtype.name)
else: # create a numpy array
try:
import numpy as np
except ImportError as exc:
raise ImportError(
"Converting binary to numpy.ndarray requires numpy to be installed."
) from exc
if dtype == BinaryVectorDtype.INT8:
data = np.frombuffer(self[offset:], dtype="int8")
elif dtype == BinaryVectorDtype.FLOAT32:
if n_bytes % 4:
raise ValueError(
"Corrupt data. N bytes for a float32 vector must be a multiple of 4."
)
data = np.frombuffer(self[offset:], dtype="float32")
elif dtype == BinaryVectorDtype.PACKED_BIT:
# data packed as uint8
if padding and not n_bytes:
raise ValueError("Corrupt data. Vector has a padding P, but no data.")
if padding > 7 or padding < 0:
raise ValueError(f"Corrupt data. Padding ({padding}) must be between 0 and 7.")
data = np.frombuffer(self[offset:], dtype="uint8")
if padding and np.unpackbits(data[-1])[-padding:].sum() > 0:
warnings.warn(
"Vector has a padding P, but bits in the final byte lower than P are non-zero. For pymongo>=5.0, they must be zero.",
DeprecationWarning,
stacklevel=2,
)
else:
raise NotImplementedError("Binary Vector dtype %s not yet supported" % dtype.name)
return BinaryVector(data, dtype, padding)
@property @property
def subtype(self) -> int: def subtype(self):
"""Subtype of this binary data.""" """Subtype of this binary data.
"""
return self.__subtype return self.__subtype
def __getnewargs__(self) -> Tuple[bytes, int]: # type: ignore[override] def __getnewargs__(self):
# Work around http://bugs.python.org/issue7382 # Work around http://bugs.python.org/issue7382
data = super().__getnewargs__()[0] data = super(Binary, self).__getnewargs__()[0]
if not isinstance(data, bytes): if PY3 and not isinstance(data, binary_type):
data = data.encode("latin-1") data = data.encode('latin-1')
return data, self.__subtype return data, self.__subtype
def __eq__(self, other: Any) -> bool: def __eq__(self, other):
if isinstance(other, Binary): if isinstance(other, Binary):
return (self.__subtype, bytes(self)) == (other.subtype, bytes(other)) return ((self.__subtype, binary_type(self)) ==
(other.subtype, binary_type(other)))
# We don't return NotImplemented here because if we did then # We don't return NotImplemented here because if we did then
# Binary("foo") == "foo" would return True, since Binary is a # Binary("foo") == "foo" would return True, since Binary is a
# subclass of str... # subclass of str...
return False return False
def __hash__(self) -> int: def __hash__(self):
return super().__hash__() ^ hash(self.__subtype) return super(Binary, self).__hash__() ^ hash(self.__subtype)
def __ne__(self, other: Any) -> bool: def __ne__(self, other):
return not self == other return not self == other
def __repr__(self) -> str: def __repr__(self):
if self.__subtype == SENSITIVE_SUBTYPE: return "Binary(%s, %s)" % (binary_type.__repr__(self), self.__subtype)
return f"<Binary(REDACTED, {self.__subtype})>"
else:
return f"Binary({bytes.__repr__(self)}, {self.__subtype})" class UUIDLegacy(Binary):
"""UUID wrapper to support working with UUIDs stored as legacy
BSON binary subtype 3.
.. doctest::
>>> import uuid
>>> from bson.binary import Binary, UUIDLegacy, UUID_SUBTYPE
>>> my_uuid = uuid.uuid4()
>>> coll = db.test
>>> coll.uuid_subtype = UUID_SUBTYPE
>>> coll.insert({'uuid': Binary(my_uuid.bytes, 3)})
ObjectId('...')
>>> coll.find({'uuid': my_uuid}).count()
0
>>> coll.find({'uuid': UUIDLegacy(my_uuid)}).count()
1
>>> coll.find({'uuid': UUIDLegacy(my_uuid)})[0]['uuid']
UUID('...')
>>>
>>> # Convert from subtype 3 to subtype 4
>>> doc = coll.find_one({'uuid': UUIDLegacy(my_uuid)})
>>> coll.save(doc)
ObjectId('...')
>>> coll.find({'uuid': UUIDLegacy(my_uuid)}).count()
0
>>> coll.find({'uuid': {'$in': [UUIDLegacy(my_uuid), my_uuid]}}).count()
1
>>> coll.find_one({'uuid': my_uuid})['uuid']
UUID('...')
Raises TypeError if `obj` is not an instance of :class:`~uuid.UUID`.
:Parameters:
- `obj`: An instance of :class:`~uuid.UUID`.
"""
def __new__(cls, obj):
if not isinstance(obj, UUID):
raise TypeError("obj must be an instance of uuid.UUID")
# Python 3.0(.1) returns a bytearray instance for bytes (3.1 and
# newer just return a bytes instance). Convert that to binary_type
# for compatibility.
self = Binary.__new__(cls, binary_type(obj.bytes), OLD_UUID_SUBTYPE)
self.__uuid = obj
return self
def __getnewargs__(self):
# Support copy and deepcopy
return (self.__uuid,)
@property
def uuid(self):
"""UUID instance wrapped by this UUIDLegacy instance.
"""
return self.__uuid
def __repr__(self):
return "UUIDLegacy('%s')" % self.__uuid

View File

@ -1,233 +0,0 @@
/*
* Copyright 2013-2016 MongoDB, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
#ifndef BSON_ENDIAN_H
#define BSON_ENDIAN_H
#if defined(__sun)
# include <sys/byteorder.h>
#endif
#ifdef _MSC_VER
# define BSON_INLINE __inline
#else
# include <stdint.h>
# define BSON_INLINE __inline__
#endif
#define BSON_BIG_ENDIAN 4321
#define BSON_LITTLE_ENDIAN 1234
/* WORDS_BIGENDIAN from pyconfig.h / Python.h */
#ifdef WORDS_BIGENDIAN
# define BSON_BYTE_ORDER BSON_BIG_ENDIAN
#else
# define BSON_BYTE_ORDER BSON_LITTLE_ENDIAN
#endif
#if defined(__sun)
# define BSON_UINT16_SWAP_LE_BE(v) BSWAP_16((uint16_t)v)
# define BSON_UINT32_SWAP_LE_BE(v) BSWAP_32((uint32_t)v)
# define BSON_UINT64_SWAP_LE_BE(v) BSWAP_64((uint64_t)v)
#elif defined(__clang__) && defined(__clang_major__) && defined(__clang_minor__) && \
(__clang_major__ >= 3) && (__clang_minor__ >= 1)
# if __has_builtin(__builtin_bswap16)
# define BSON_UINT16_SWAP_LE_BE(v) __builtin_bswap16(v)
# endif
# if __has_builtin(__builtin_bswap32)
# define BSON_UINT32_SWAP_LE_BE(v) __builtin_bswap32(v)
# endif
# if __has_builtin(__builtin_bswap64)
# define BSON_UINT64_SWAP_LE_BE(v) __builtin_bswap64(v)
# endif
#elif defined(__GNUC__) && (__GNUC__ >= 4)
# if __GNUC__ >= 4 && defined (__GNUC_MINOR__) && __GNUC_MINOR__ >= 3
# define BSON_UINT32_SWAP_LE_BE(v) __builtin_bswap32 ((uint32_t)v)
# define BSON_UINT64_SWAP_LE_BE(v) __builtin_bswap64 ((uint64_t)v)
# endif
# if __GNUC__ >= 4 && defined (__GNUC_MINOR__) && __GNUC_MINOR__ >= 8
# define BSON_UINT16_SWAP_LE_BE(v) __builtin_bswap16 ((uint32_t)v)
# endif
#endif
#ifndef BSON_UINT16_SWAP_LE_BE
# define BSON_UINT16_SWAP_LE_BE(v) __bson_uint16_swap_slow ((uint16_t)v)
#endif
#ifndef BSON_UINT32_SWAP_LE_BE
# define BSON_UINT32_SWAP_LE_BE(v) __bson_uint32_swap_slow ((uint32_t)v)
#endif
#ifndef BSON_UINT64_SWAP_LE_BE
# define BSON_UINT64_SWAP_LE_BE(v) __bson_uint64_swap_slow ((uint64_t)v)
#endif
#if BSON_BYTE_ORDER == BSON_LITTLE_ENDIAN
# define BSON_UINT16_FROM_LE(v) ((uint16_t)v)
# define BSON_UINT16_TO_LE(v) ((uint16_t)v)
# define BSON_UINT16_FROM_BE(v) BSON_UINT16_SWAP_LE_BE (v)
# define BSON_UINT16_TO_BE(v) BSON_UINT16_SWAP_LE_BE (v)
# define BSON_UINT32_FROM_LE(v) ((uint32_t)v)
# define BSON_UINT32_TO_LE(v) ((uint32_t)v)
# define BSON_UINT32_FROM_BE(v) BSON_UINT32_SWAP_LE_BE (v)
# define BSON_UINT32_TO_BE(v) BSON_UINT32_SWAP_LE_BE (v)
# define BSON_UINT64_FROM_LE(v) ((uint64_t)v)
# define BSON_UINT64_TO_LE(v) ((uint64_t)v)
# define BSON_UINT64_FROM_BE(v) BSON_UINT64_SWAP_LE_BE (v)
# define BSON_UINT64_TO_BE(v) BSON_UINT64_SWAP_LE_BE (v)
# define BSON_DOUBLE_FROM_LE(v) ((double)v)
# define BSON_DOUBLE_TO_LE(v) ((double)v)
#elif BSON_BYTE_ORDER == BSON_BIG_ENDIAN
# define BSON_UINT16_FROM_LE(v) BSON_UINT16_SWAP_LE_BE (v)
# define BSON_UINT16_TO_LE(v) BSON_UINT16_SWAP_LE_BE (v)
# define BSON_UINT16_FROM_BE(v) ((uint16_t)v)
# define BSON_UINT16_TO_BE(v) ((uint16_t)v)
# define BSON_UINT32_FROM_LE(v) BSON_UINT32_SWAP_LE_BE (v)
# define BSON_UINT32_TO_LE(v) BSON_UINT32_SWAP_LE_BE (v)
# define BSON_UINT32_FROM_BE(v) ((uint32_t)v)
# define BSON_UINT32_TO_BE(v) ((uint32_t)v)
# define BSON_UINT64_FROM_LE(v) BSON_UINT64_SWAP_LE_BE (v)
# define BSON_UINT64_TO_LE(v) BSON_UINT64_SWAP_LE_BE (v)
# define BSON_UINT64_FROM_BE(v) ((uint64_t)v)
# define BSON_UINT64_TO_BE(v) ((uint64_t)v)
# define BSON_DOUBLE_FROM_LE(v) (__bson_double_swap_slow (v))
# define BSON_DOUBLE_TO_LE(v) (__bson_double_swap_slow (v))
#else
# error "The endianness of target architecture is unknown."
#endif
/*
*--------------------------------------------------------------------------
*
* __bson_uint16_swap_slow --
*
* Fallback endianness conversion for 16-bit integers.
*
* Returns:
* The endian swapped version.
*
* Side effects:
* None.
*
*--------------------------------------------------------------------------
*/
static BSON_INLINE uint16_t
__bson_uint16_swap_slow (uint16_t v) /* IN */
{
return ((v & 0x00FF) << 8) |
((v & 0xFF00) >> 8);
}
/*
*--------------------------------------------------------------------------
*
* __bson_uint32_swap_slow --
*
* Fallback endianness conversion for 32-bit integers.
*
* Returns:
* The endian swapped version.
*
* Side effects:
* None.
*
*--------------------------------------------------------------------------
*/
static BSON_INLINE uint32_t
__bson_uint32_swap_slow (uint32_t v) /* IN */
{
return ((v & 0x000000FFU) << 24) |
((v & 0x0000FF00U) << 8) |
((v & 0x00FF0000U) >> 8) |
((v & 0xFF000000U) >> 24);
}
/*
*--------------------------------------------------------------------------
*
* __bson_uint64_swap_slow --
*
* Fallback endianness conversion for 64-bit integers.
*
* Returns:
* The endian swapped version.
*
* Side effects:
* None.
*
*--------------------------------------------------------------------------
*/
static BSON_INLINE uint64_t
__bson_uint64_swap_slow (uint64_t v) /* IN */
{
return ((v & 0x00000000000000FFULL) << 56) |
((v & 0x000000000000FF00ULL) << 40) |
((v & 0x0000000000FF0000ULL) << 24) |
((v & 0x00000000FF000000ULL) << 8) |
((v & 0x000000FF00000000ULL) >> 8) |
((v & 0x0000FF0000000000ULL) >> 24) |
((v & 0x00FF000000000000ULL) >> 40) |
((v & 0xFF00000000000000ULL) >> 56);
}
/*
*--------------------------------------------------------------------------
*
* __bson_double_swap_slow --
*
* Fallback endianness conversion for double floating point.
*
* Returns:
* The endian swapped version.
*
* Side effects:
* None.
*
*--------------------------------------------------------------------------
*/
static BSON_INLINE double
__bson_double_swap_slow (double v) /* IN */
{
uint64_t uv;
memcpy(&uv, &v, sizeof(v));
uv = BSON_UINT64_SWAP_LE_BE(uv);
memcpy(&v, &uv, sizeof(v));
return v;
}
#endif /* BSON_ENDIAN_H */

View File

@ -14,10 +14,6 @@
* limitations under the License. * limitations under the License.
*/ */
/* Include Python.h so we can set Python's error indicator. */
#define PY_SSIZE_T_CLEAN
#include "Python.h"
#include <stdlib.h> #include <stdlib.h>
#include <string.h> #include <string.h>
@ -31,19 +27,12 @@ struct buffer {
int position; int position;
}; };
/* Set Python's error indicator to MemoryError.
* Called after allocation failures. */
static void set_memory_error(void) {
PyErr_NoMemory();
}
/* Allocate and return a new buffer. /* Allocate and return a new buffer.
* Return NULL and sets MemoryError on allocation failure. */ * Return NULL on allocation failure. */
buffer_t pymongo_buffer_new(void) { buffer_t buffer_new(void) {
buffer_t buffer; buffer_t buffer;
buffer = (buffer_t)malloc(sizeof(struct buffer)); buffer = (buffer_t)malloc(sizeof(struct buffer));
if (buffer == NULL) { if (buffer == NULL) {
set_memory_error();
return NULL; return NULL;
} }
@ -52,7 +41,6 @@ buffer_t pymongo_buffer_new(void) {
buffer->buffer = (char*)malloc(sizeof(char) * INITIAL_BUFFER_SIZE); buffer->buffer = (char*)malloc(sizeof(char) * INITIAL_BUFFER_SIZE);
if (buffer->buffer == NULL) { if (buffer->buffer == NULL) {
free(buffer); free(buffer);
set_memory_error();
return NULL; return NULL;
} }
@ -61,20 +49,17 @@ buffer_t pymongo_buffer_new(void) {
/* Free the memory allocated for `buffer`. /* Free the memory allocated for `buffer`.
* Return non-zero on failure. */ * Return non-zero on failure. */
int pymongo_buffer_free(buffer_t buffer) { int buffer_free(buffer_t buffer) {
if (buffer == NULL) { if (buffer == NULL) {
return 1; return 1;
} }
/* Buffer will be NULL when buffer_grow fails. */ free(buffer->buffer);
if (buffer->buffer != NULL) {
free(buffer->buffer);
}
free(buffer); free(buffer);
return 0; return 0;
} }
/* Grow `buffer` to at least `min_length`. /* Grow `buffer` to at least `min_length`.
* Return non-zero and sets MemoryError on allocation failure. */ * Return non-zero on allocation failure. */
static int buffer_grow(buffer_t buffer, int min_length) { static int buffer_grow(buffer_t buffer, int min_length) {
int old_size = 0; int old_size = 0;
int size = buffer->size; int size = buffer->size;
@ -94,7 +79,7 @@ static int buffer_grow(buffer_t buffer, int min_length) {
buffer->buffer = (char*)realloc(buffer->buffer, sizeof(char) * size); buffer->buffer = (char*)realloc(buffer->buffer, sizeof(char) * size);
if (buffer->buffer == NULL) { if (buffer->buffer == NULL) {
free(old_buffer); free(old_buffer);
set_memory_error(); free(buffer);
return 1; return 1;
} }
buffer->size = size; buffer->size = size;
@ -102,27 +87,17 @@ static int buffer_grow(buffer_t buffer, int min_length) {
} }
/* Assure that `buffer` has at least `size` free bytes (and grow if needed). /* Assure that `buffer` has at least `size` free bytes (and grow if needed).
* Return non-zero and sets MemoryError on allocation failure. * Return non-zero on allocation failure. */
* Return non-zero and sets ValueError if `size` would exceed 2GiB. */
static int buffer_assure_space(buffer_t buffer, int size) { static int buffer_assure_space(buffer_t buffer, int size) {
int new_size = buffer->position + size; if (buffer->position + size <= buffer->size) {
/* Check for overflow. */
if (new_size < buffer->position) {
PyErr_SetString(PyExc_ValueError,
"Document would overflow BSON size limit");
return 1;
}
if (new_size <= buffer->size) {
return 0; return 0;
} }
return buffer_grow(buffer, new_size); return buffer_grow(buffer, buffer->position + size);
} }
/* Save `size` bytes from the current position in `buffer` (and grow if needed). /* Save `size` bytes from the current position in `buffer` (and grow if needed).
* Return offset for writing, or -1 on failure. * Return offset for writing, or -1 on allocation failure. */
* Sets MemoryError or ValueError on failure. */ buffer_position buffer_save_space(buffer_t buffer, int size) {
buffer_position pymongo_buffer_save_space(buffer_t buffer, int size) {
int position = buffer->position; int position = buffer->position;
if (buffer_assure_space(buffer, size) != 0) { if (buffer_assure_space(buffer, size) != 0) {
return -1; return -1;
@ -132,9 +107,8 @@ buffer_position pymongo_buffer_save_space(buffer_t buffer, int size) {
} }
/* Write `size` bytes from `data` to `buffer` (and grow if needed). /* Write `size` bytes from `data` to `buffer` (and grow if needed).
* Return non-zero on failure. * Return non-zero on allocation failure. */
* Sets MemoryError or ValueError on failure. */ int buffer_write(buffer_t buffer, const char* data, int size) {
int pymongo_buffer_write(buffer_t buffer, const char* data, int size) {
if (buffer_assure_space(buffer, size) != 0) { if (buffer_assure_space(buffer, size) != 0) {
return 1; return 1;
} }
@ -144,14 +118,29 @@ int pymongo_buffer_write(buffer_t buffer, const char* data, int size) {
return 0; return 0;
} }
int pymongo_buffer_get_position(buffer_t buffer) { /* Write `size` bytes from `data` to `buffer` at position `position`.
* Does not change the internal position of `buffer`.
* Return non-zero if buffer isn't large enough for write. */
int buffer_write_at_position(buffer_t buffer, buffer_position position,
const char* data, int size) {
if (position + size > buffer->size) {
buffer_free(buffer);
return 1;
}
memcpy(buffer->buffer + position, data, size);
return 0;
}
int buffer_get_position(buffer_t buffer) {
return buffer->position; return buffer->position;
} }
char* pymongo_buffer_get_buffer(buffer_t buffer) { char* buffer_get_buffer(buffer_t buffer) {
return buffer->buffer; return buffer->buffer;
} }
void pymongo_buffer_update_position(buffer_t buffer, buffer_position new_position) { void buffer_update_position(buffer_t buffer, buffer_position new_position) {
buffer->position = new_position; buffer->position = new_position;
} }

View File

@ -27,25 +27,30 @@ typedef int buffer_position;
/* Allocate and return a new buffer. /* Allocate and return a new buffer.
* Return NULL on allocation failure. */ * Return NULL on allocation failure. */
buffer_t pymongo_buffer_new(void); buffer_t buffer_new(void);
/* Free the memory allocated for `buffer`. /* Free the memory allocated for `buffer`.
* Return non-zero on failure. */ * Return non-zero on failure. */
int pymongo_buffer_free(buffer_t buffer); int buffer_free(buffer_t buffer);
/* Save `size` bytes from the current position in `buffer` (and grow if needed). /* Save `size` bytes from the current position in `buffer` (and grow if needed).
* Return offset for writing, or -1 on allocation failure. */ * Return offset for writing, or -1 on allocation failure. */
buffer_position pymongo_buffer_save_space(buffer_t buffer, int size); buffer_position buffer_save_space(buffer_t buffer, int size);
/* Write `size` bytes from `data` to `buffer` (and grow if needed). /* Write `size` bytes from `data` to `buffer` (and grow if needed).
* Return non-zero on allocation failure. */ * Return non-zero on allocation failure. */
int pymongo_buffer_write(buffer_t buffer, const char* data, int size); int buffer_write(buffer_t buffer, const char* data, int size);
/* Write `size` bytes from `data` to `buffer` at position `position`.
* Does not change the internal position of `buffer`.
* Return non-zero if buffer isn't large enough for write. */
int buffer_write_at_position(buffer_t buffer, buffer_position position, const char* data, int size);
/* Getters for the internals of a buffer_t. /* Getters for the internals of a buffer_t.
* Should try to avoid using these as much as possible * Should try to avoid using these as much as possible
* since they break the abstraction. */ * since they break the abstraction. */
buffer_position pymongo_buffer_get_position(buffer_t buffer); buffer_position buffer_get_position(buffer_t buffer);
char* pymongo_buffer_get_buffer(buffer_t buffer); char* buffer_get_buffer(buffer_t buffer);
void pymongo_buffer_update_position(buffer_t buffer, buffer_position new_position); void buffer_update_position(buffer_t buffer, buffer_position new_position);
#endif #endif

View File

@ -1,4 +1,4 @@
# Copyright 2009-present MongoDB, Inc. # Copyright 2009-2015 MongoDB, Inc.
# #
# Licensed under the Apache License, Version 2.0 (the "License"); # Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License. # you may not use this file except in compliance with the License.
@ -12,89 +12,71 @@
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
"""Tools for representing JavaScript code in BSON.""" """Tools for representing JavaScript code in BSON.
from __future__ import annotations """
from collections.abc import Mapping as _Mapping
from typing import Any, Mapping, Optional, Type, Union
class Code(str): class Code(str):
"""BSON's JavaScript code type. """BSON's JavaScript code type.
Raises :class:`TypeError` if `code` is not an instance of Raises :class:`TypeError` if `code` is not an instance of
:class:`str` or `scope` is not ``None`` or an instance :class:`basestring` (:class:`str` in python 3) or `scope`
of :class:`dict`. is not ``None`` or an instance of :class:`dict`.
Scope variables can be set by passing a dictionary as the `scope` Scope variables can be set by passing a dictionary as the `scope`
argument or by using keyword arguments. If a variable is set as a argument or by using keyword arguments. If a variable is set as a
keyword argument it will override any setting for that variable in keyword argument it will override any setting for that variable in
the `scope` dictionary. the `scope` dictionary.
:param code: A string containing JavaScript code to be evaluated or another :Parameters:
instance of Code. In the latter case, the scope of `code` becomes this - `code`: string containing JavaScript code to be evaluated
Code's :attr:`scope`. - `scope` (optional): dictionary representing the scope in which
:param scope: dictionary representing the scope in which
`code` should be evaluated - a mapping from identifiers (as `code` should be evaluated - a mapping from identifiers (as
strings) to values. Defaults to ``None``. This is applied after any strings) to values
scope associated with a given `code` above. - `**kwargs` (optional): scope variables can also be passed as
:param kwargs: scope variables can also be passed as keyword arguments
keyword arguments. These are applied after `scope` and `code`.
.. versionchanged:: 3.4
The default value for :attr:`scope` is ``None`` instead of ``{}``.
.. versionadded:: 1.9
Ability to pass scope values using keyword arguments.
""" """
_type_marker = 13 _type_marker = 13
__scope: Union[Mapping[str, Any], None]
def __new__( def __new__(cls, code, scope=None, **kwargs):
cls: Type[Code], if not isinstance(code, basestring):
code: Union[str, Code], raise TypeError("code must be an "
scope: Optional[Mapping[str, Any]] = None, "instance of %s" % (basestring.__name__,))
**kwargs: Any,
) -> Code:
if not isinstance(code, str):
raise TypeError(f"code must be an instance of str, not {type(code)}")
self = str.__new__(cls, code) self = str.__new__(cls, code)
try: try:
self.__scope = code.scope # type: ignore self.__scope = code.scope
except AttributeError: except AttributeError:
self.__scope = None self.__scope = {}
if scope is not None: if scope is not None:
if not isinstance(scope, _Mapping): if not isinstance(scope, dict):
raise TypeError(f"scope must be an instance of dict, not {type(scope)}") raise TypeError("scope must be an instance of dict")
if self.__scope is not None: self.__scope.update(scope)
self.__scope.update(scope) # type: ignore
else:
self.__scope = scope
if kwargs: self.__scope.update(kwargs)
if self.__scope is not None:
self.__scope.update(kwargs) # type: ignore
else:
self.__scope = kwargs
return self return self
@property @property
def scope(self) -> Optional[Mapping[str, Any]]: def scope(self):
"""Scope dictionary for this instance or ``None``.""" """Scope dictionary for this instance.
"""
return self.__scope return self.__scope
def __repr__(self) -> str: def __repr__(self):
return f"Code({str.__repr__(self)}, {self.__scope!r})" return "Code(%s, %r)" % (str.__repr__(self), self.__scope)
def __eq__(self, other: Any) -> bool: def __eq__(self, other):
if isinstance(other, Code): if isinstance(other, Code):
return (self.__scope, str(self)) == (other.__scope, str(other)) return (self.__scope, str(self)) == (other.__scope, str(other))
return False return False
__hash__: Any = None __hash__ = None
def __ne__(self, other: Any) -> bool: def __ne__(self, other):
return not self == other return not self == other

View File

@ -1,4 +1,4 @@
# Copyright 2014-present MongoDB, Inc. # Copyright 2014-2015 MongoDB, Inc.
# #
# Licensed under the Apache License, Version 2.0 (the "License"); # Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License. # you may not use this file except in compliance with the License.
@ -13,493 +13,65 @@
# limitations under the License. # limitations under the License.
"""Tools for specifying BSON codec options.""" """Tools for specifying BSON codec options."""
from __future__ import annotations
import abc from bson.binary import (ALL_UUID_REPRESENTATIONS,
import datetime PYTHON_LEGACY,
import enum UUID_REPRESENTATION_NAMES)
from collections.abc import MutableMapping as _MutableMapping
from typing import (
TYPE_CHECKING,
Any,
Callable,
Generic,
Iterable,
Mapping,
NamedTuple,
Optional,
Tuple,
Type,
Union,
cast,
)
from bson.binary import (
ALL_UUID_REPRESENTATIONS,
UUID_REPRESENTATION_NAMES,
UuidRepresentation,
)
from bson.typings import _DocumentType
_RAW_BSON_DOCUMENT_MARKER = 101
def _raw_document_class(document_class: Any) -> bool: class CodecOptions(tuple):
"""Determine if a document_class is a RawBSONDocument class.""" """Encapsulates BSON options used in CRUD operations.
marker = getattr(document_class, "_type_marker", None)
return marker == _RAW_BSON_DOCUMENT_MARKER
:Parameters:
class TypeEncoder(abc.ABC): - `document_class`: BSON documents returned in queries will be decoded
"""Base class for defining type codec classes which describe how a to an instance of this class.
custom type can be transformed to one of the types BSON understands. - `tz_aware`: If ``True``, BSON datetimes will be decoded to timezone
aware instances of :class:`~datetime.datetime`. Otherwise they will be
Codec classes must implement the ``python_type`` attribute, and the naive. Defaults to ``False``.
``transform_python`` method to support encoding. - `uuid_representation`: The BSON representation to use when encoding
and decoding instances of :class:`~uuid.UUID`. Defaults to
See `encode data with type codecs <https://www.mongodb.com/docs/languages/python/pymongo-driver/current/data-formats/custom-types/type-codecs/#encode-data-with-type-codecs>`_ documentation for an example. :data:`~bson.binary.PYTHON_LEGACY`.
""" """
@abc.abstractproperty __slots__ = ()
def python_type(self) -> Any:
"""The Python type to be converted into something serializable.""" def __new__(cls, document_class=dict,
tz_aware=False, uuid_representation=PYTHON_LEGACY):
@abc.abstractmethod if not isinstance(tz_aware, bool):
def transform_python(self, value: Any) -> Any: raise TypeError("tz_aware must be True or False")
"""Convert the given Python object into something serializable.""" if uuid_representation not in ALL_UUID_REPRESENTATIONS:
raise ValueError("uuid_representation must be a value "
"from bson.binary.ALL_UUID_REPRESENTATIONS")
class TypeDecoder(abc.ABC):
"""Base class for defining type codec classes which describe how a return tuple.__new__(
BSON type can be transformed to a custom type. cls, (document_class, tz_aware, uuid_representation))
Codec classes must implement the ``bson_type`` attribute, and the def __repr__(self):
``transform_bson`` method to support decoding. if self.document_class is dict:
document_class_repr = 'dict'
See `encode data with type codecs <https://www.mongodb.com/docs/languages/python/pymongo-driver/current/data-formats/custom-types/type-codecs/#encode-data-with-type-codecs>`_ documentation for an example.
"""
@abc.abstractproperty
def bson_type(self) -> Any:
"""The BSON type to be converted into our own type."""
@abc.abstractmethod
def transform_bson(self, value: Any) -> Any:
"""Convert the given BSON value into our own type."""
class TypeCodec(TypeEncoder, TypeDecoder):
"""Base class for defining type codec classes which describe how a
custom type can be transformed to/from one of the types :mod:`bson`
can already encode/decode.
Codec classes must implement the ``python_type`` attribute, and the
``transform_python`` method to support encoding, as well as the
``bson_type`` attribute, and the ``transform_bson`` method to support
decoding.
See `encode data with type codecs <https://www.mongodb.com/docs/languages/python/pymongo-driver/current/data-formats/custom-types/type-codecs/#encode-data-with-type-codecs>`_ documentation for an example.
"""
_Codec = Union[TypeEncoder, TypeDecoder, TypeCodec]
_Fallback = Callable[[Any], Any]
class TypeRegistry:
"""Encapsulates type codecs used in encoding and / or decoding BSON, as
well as the fallback encoder. Type registries cannot be modified after
instantiation.
``TypeRegistry`` can be initialized with an iterable of type codecs, and
a callable for the fallback encoder::
>>> from bson.codec_options import TypeRegistry
>>> type_registry = TypeRegistry([Codec1, Codec2, Codec3, ...],
... fallback_encoder)
See `add codec to the type registry <https://www.mongodb.com/docs/languages/python/pymongo-driver/current/data-formats/custom-types/type-codecs/#add-codec-to-the-type-registry>`_ documentation for an example.
:param type_codecs: iterable of type codec instances. If
``type_codecs`` contains multiple codecs that transform a single
python or BSON type, the transformation specified by the type codec
occurring last prevails. A TypeError will be raised if one or more
type codecs modify the encoding behavior of a built-in :mod:`bson`
type.
:param fallback_encoder: callable that accepts a single,
unencodable python value and transforms it into a type that
:mod:`bson` can encode. See `define a fallback encoder <https://www.mongodb.com/docs/languages/python/pymongo-driver/current/data-formats/custom-types/type-codecs/#define-a-fallback-encoder>`_
documentation for an example.
"""
def __init__(
self,
type_codecs: Optional[Iterable[_Codec]] = None,
fallback_encoder: Optional[_Fallback] = None,
) -> None:
self.__type_codecs = list(type_codecs or [])
self._fallback_encoder = fallback_encoder
self._encoder_map: dict[Any, Any] = {}
self._decoder_map: dict[Any, Any] = {}
if self._fallback_encoder is not None:
if not callable(fallback_encoder):
raise TypeError("fallback_encoder %r is not a callable" % (fallback_encoder))
for codec in self.__type_codecs:
is_valid_codec = False
if isinstance(codec, TypeEncoder):
self._validate_type_encoder(codec)
is_valid_codec = True
self._encoder_map[codec.python_type] = codec.transform_python
if isinstance(codec, TypeDecoder):
is_valid_codec = True
self._decoder_map[codec.bson_type] = codec.transform_bson
if not is_valid_codec:
raise TypeError(
f"Expected an instance of {TypeEncoder.__name__}, {TypeDecoder.__name__}, or {TypeCodec.__name__}, got {codec!r} instead"
)
@property
def codecs(self) -> list[TypeEncoder | TypeDecoder | TypeCodec]:
"""The list of type codecs in this registry."""
return self.__type_codecs
@property
def fallback_encoder(self) -> Optional[_Fallback]:
"""The fallback encoder in this registry."""
return self._fallback_encoder
def _validate_type_encoder(self, codec: _Codec) -> None:
from bson import _BUILT_IN_TYPES
for pytype in _BUILT_IN_TYPES:
if issubclass(cast(TypeCodec, codec).python_type, pytype):
err_msg = (
"TypeEncoders cannot change how built-in types are "
f"encoded (encoder {codec} transforms type {pytype})"
)
raise TypeError(err_msg)
def __repr__(self) -> str:
return "{}(type_codecs={!r}, fallback_encoder={!r})".format(
self.__class__.__name__,
self.__type_codecs,
self._fallback_encoder,
)
def __eq__(self, other: Any) -> Any:
if not isinstance(other, type(self)):
return NotImplemented
return (
(self._decoder_map == other._decoder_map)
and (self._encoder_map == other._encoder_map)
and (self._fallback_encoder == other._fallback_encoder)
)
class DatetimeConversion(int, enum.Enum):
"""Options for decoding BSON datetimes."""
DATETIME = 1
"""Decode a BSON UTC datetime as a :class:`datetime.datetime`.
BSON UTC datetimes that cannot be represented as a
:class:`~datetime.datetime` will raise an :class:`OverflowError`
or a :class:`ValueError`.
.. versionadded 4.3
"""
DATETIME_CLAMP = 2
"""Decode a BSON UTC datetime as a :class:`datetime.datetime`, clamping
to :attr:`~datetime.datetime.min` and :attr:`~datetime.datetime.max`.
.. versionadded 4.3
"""
DATETIME_MS = 3
"""Decode a BSON UTC datetime as a :class:`~bson.datetime_ms.DatetimeMS`
object.
.. versionadded 4.3
"""
DATETIME_AUTO = 4
"""Decode a BSON UTC datetime as a :class:`datetime.datetime` if possible,
and a :class:`~bson.datetime_ms.DatetimeMS` if not.
.. versionadded 4.3
"""
class _BaseCodecOptions(NamedTuple):
document_class: Type[Mapping[str, Any]]
tz_aware: bool
uuid_representation: int
unicode_decode_error_handler: str
tzinfo: Optional[datetime.tzinfo]
type_registry: TypeRegistry
datetime_conversion: Optional[DatetimeConversion]
if TYPE_CHECKING:
class CodecOptions(Tuple[_DocumentType], Generic[_DocumentType]):
document_class: Type[_DocumentType]
tz_aware: bool
uuid_representation: int
unicode_decode_error_handler: Optional[str]
tzinfo: Optional[datetime.tzinfo]
type_registry: TypeRegistry
datetime_conversion: Optional[int]
def __new__(
cls: Type[CodecOptions[_DocumentType]],
document_class: Optional[Type[_DocumentType]] = ...,
tz_aware: bool = ...,
uuid_representation: Optional[int] = ...,
unicode_decode_error_handler: Optional[str] = ...,
tzinfo: Optional[datetime.tzinfo] = ...,
type_registry: Optional[TypeRegistry] = ...,
datetime_conversion: Optional[int] = ...,
) -> CodecOptions[_DocumentType]:
...
# CodecOptions API
def with_options(self, **kwargs: Any) -> CodecOptions[Any]:
...
def _arguments_repr(self) -> str:
...
# NamedTuple API
@classmethod
def _make(cls, obj: Iterable[Any]) -> CodecOptions[_DocumentType]:
...
def _asdict(self) -> dict[str, Any]:
...
def _replace(self, **kwargs: Any) -> CodecOptions[_DocumentType]:
...
_source: str
_fields: Tuple[str]
else:
class CodecOptions(_BaseCodecOptions):
"""Encapsulates options used encoding and / or decoding BSON."""
def __init__(self, *args, **kwargs):
"""Encapsulates options used encoding and / or decoding BSON.
The `document_class` option is used to define a custom type for use
decoding BSON documents. Access to the underlying raw BSON bytes for
a document is available using the :class:`~bson.raw_bson.RawBSONDocument`
type::
>>> from bson.raw_bson import RawBSONDocument
>>> from bson.codec_options import CodecOptions
>>> codec_options = CodecOptions(document_class=RawBSONDocument)
>>> coll = db.get_collection('test', codec_options=codec_options)
>>> doc = coll.find_one()
>>> doc.raw
'\\x16\\x00\\x00\\x00\\x07_id\\x00[0\\x165\\x91\\x10\\xea\\x14\\xe8\\xc5\\x8b\\x93\\x00'
The document class can be any type that inherits from
:class:`~collections.abc.MutableMapping`::
>>> class AttributeDict(dict):
... # A dict that supports attribute access.
... def __getattr__(self, key):
... return self[key]
... def __setattr__(self, key, value):
... self[key] = value
...
>>> codec_options = CodecOptions(document_class=AttributeDict)
>>> coll = db.get_collection('test', codec_options=codec_options)
>>> doc = coll.find_one()
>>> doc._id
ObjectId('5b3016359110ea14e8c58b93')
See `Dates and Times <https://www.mongodb.com/docs/languages/python/pymongo-driver/current/data-formats/dates-and-times/#dates-and-times>`_ for examples using the `tz_aware` and
`tzinfo` options.
See `UUID <https://www.mongodb.com/docs/languages/python/pymongo-driver/current/data-formats/uuid/#universally-unique-ids--uuids->`_ for examples using the `uuid_representation`
option.
:param document_class: BSON documents returned in queries will be decoded
to an instance of this class. Must be a subclass of
:class:`~collections.abc.MutableMapping`. Defaults to :class:`dict`.
:param tz_aware: If ``True``, BSON datetimes will be decoded to timezone
aware instances of :class:`~datetime.datetime`. Otherwise they will be
naive. Defaults to ``False``.
:param uuid_representation: The BSON representation to use when encoding
and decoding instances of :class:`~uuid.UUID`. Defaults to
:data:`~bson.binary.UuidRepresentation.UNSPECIFIED`. New
applications should consider setting this to
:data:`~bson.binary.UuidRepresentation.STANDARD` for cross language
compatibility. See `UUID representations <https://www.mongodb.com/docs/languages/python/pymongo-driver/current/data-formats/uuid/#universally-unique-ids--uuids->`_ for details.
:param unicode_decode_error_handler: The error handler to apply when
a Unicode-related error occurs during BSON decoding that would
otherwise raise :exc:`UnicodeDecodeError`. Valid options include
'strict', 'replace', 'backslashreplace', 'surrogateescape', and
'ignore'. Defaults to 'strict'.
:param tzinfo: A :class:`~datetime.tzinfo` subclass that specifies the
timezone to/from which :class:`~datetime.datetime` objects should be
encoded/decoded.
:param type_registry: Instance of :class:`TypeRegistry` used to customize
encoding and decoding behavior.
:param datetime_conversion: Specifies how UTC datetimes should be decoded
within BSON. Valid options include 'datetime_ms' to return as a
DatetimeMS, 'datetime' to return as a datetime.datetime and
raising a ValueError for out-of-range values, 'datetime_auto' to
return DatetimeMS objects when the underlying datetime is
out-of-range and 'datetime_clamp' to clamp to the minimum and
maximum possible datetimes. Defaults to 'datetime'.
.. versionchanged:: 4.0
The default for `uuid_representation` was changed from
:const:`~bson.binary.UuidRepresentation.PYTHON_LEGACY` to
:const:`~bson.binary.UuidRepresentation.UNSPECIFIED`.
.. versionadded:: 3.8
`type_registry` attribute.
.. warning:: Care must be taken when changing
`unicode_decode_error_handler` from its default value ('strict').
The 'replace' and 'ignore' modes should not be used when documents
retrieved from the server will be modified in the client application
and stored back to the server.
"""
super().__init__()
def __new__(
cls: Type[CodecOptions],
document_class: Optional[Type[Mapping[str, Any]]] = None,
tz_aware: bool = False,
uuid_representation: Optional[int] = UuidRepresentation.UNSPECIFIED,
unicode_decode_error_handler: str = "strict",
tzinfo: Optional[datetime.tzinfo] = None,
type_registry: Optional[TypeRegistry] = None,
datetime_conversion: Optional[DatetimeConversion] = DatetimeConversion.DATETIME,
) -> CodecOptions:
doc_class = document_class or dict
# issubclass can raise TypeError for generic aliases like SON[str, Any].
# In that case we can use the base class for the comparison.
is_mapping = False
try:
is_mapping = issubclass(doc_class, _MutableMapping)
except TypeError:
if hasattr(doc_class, "__origin__"):
is_mapping = issubclass(doc_class.__origin__, _MutableMapping)
if not (is_mapping or _raw_document_class(doc_class)):
raise TypeError(
"document_class must be dict, bson.son.SON, "
"bson.raw_bson.RawBSONDocument, or a "
"subclass of collections.abc.MutableMapping"
)
if not isinstance(tz_aware, bool):
raise TypeError(f"tz_aware must be True or False, was: tz_aware={tz_aware}")
if uuid_representation not in ALL_UUID_REPRESENTATIONS:
raise ValueError(
"uuid_representation must be a value from bson.binary.UuidRepresentation"
)
if not isinstance(unicode_decode_error_handler, str):
raise ValueError(
f"unicode_decode_error_handler must be a string, not {type(unicode_decode_error_handler)}"
)
if tzinfo is not None:
if not isinstance(tzinfo, datetime.tzinfo):
raise TypeError(
f"tzinfo must be an instance of datetime.tzinfo, not {type(tzinfo)}"
)
if not tz_aware:
raise ValueError("cannot specify tzinfo without also setting tz_aware=True")
type_registry = type_registry or TypeRegistry()
if not isinstance(type_registry, TypeRegistry):
raise TypeError(
f"type_registry must be an instance of TypeRegistry, not {type(type_registry)}"
)
return tuple.__new__(
cls,
(
doc_class,
tz_aware,
uuid_representation,
unicode_decode_error_handler,
tzinfo,
type_registry,
datetime_conversion,
),
)
def _arguments_repr(self) -> str:
"""Representation of the arguments used to create this object."""
document_class_repr = (
"dict" if self.document_class is dict else repr(self.document_class)
)
uuid_rep_repr = UUID_REPRESENTATION_NAMES.get(
self.uuid_representation, self.uuid_representation
)
return (
"document_class={}, tz_aware={!r}, uuid_representation={}, "
"unicode_decode_error_handler={!r}, tzinfo={!r}, "
"type_registry={!r}, datetime_conversion={!s}".format(
document_class_repr,
self.tz_aware,
uuid_rep_repr,
self.unicode_decode_error_handler,
self.tzinfo,
self.type_registry,
self.datetime_conversion,
)
)
def __repr__(self) -> str:
return f"{self.__class__.__name__}({self._arguments_repr()})"
def with_options(self, **kwargs: Any) -> CodecOptions:
"""Make a copy of this CodecOptions, overriding some options::
>>> from bson.codec_options import DEFAULT_CODEC_OPTIONS
>>> DEFAULT_CODEC_OPTIONS.tz_aware
False
>>> options = DEFAULT_CODEC_OPTIONS.with_options(tz_aware=True)
>>> options.tz_aware
True
.. versionadded:: 3.5
"""
opts = self._asdict()
opts.update(kwargs)
return CodecOptions(**opts)
DEFAULT_CODEC_OPTIONS: CodecOptions[dict[str, Any]] = CodecOptions()
def _parse_codec_options(options: Any) -> CodecOptions[Any]:
"""Parse BSON codec options."""
kwargs = {}
for k in set(options) & {
"document_class",
"tz_aware",
"uuidrepresentation",
"unicode_decode_error_handler",
"tzinfo",
"type_registry",
"datetime_conversion",
}:
if k == "uuidrepresentation":
kwargs["uuid_representation"] = options[k]
else: else:
kwargs[k] = options[k] document_class_repr = repr(self.document_class)
return CodecOptions(**kwargs)
uuid_rep_repr = UUID_REPRESENTATION_NAMES.get(self.uuid_representation,
self.uuid_representation)
return (
'CodecOptions(document_class=%s, tz_aware=%r, uuid_representation='
'%s)' % (document_class_repr, self.tz_aware, uuid_rep_repr))
def __getnewargs__(self):
return tuple(self)
@property
def document_class(self):
return self[0]
@property
def tz_aware(self):
return self[1]
@property
def uuid_representation(self):
return self[2]
DEFAULT_CODEC_OPTIONS = CodecOptions()

View File

@ -1,182 +0,0 @@
# Copyright 2022-present MongoDB, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you
# may not use this file except in compliance with the License. You
# may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied. See the License for the specific language governing
# permissions and limitations under the License.
"""Tools for representing the BSON datetime type.
.. versionadded:: 4.3
"""
from __future__ import annotations
import calendar
import datetime
from typing import Any, Union, cast
from bson.codec_options import DEFAULT_CODEC_OPTIONS, CodecOptions, DatetimeConversion
from bson.errors import InvalidBSON
from bson.tz_util import utc
EPOCH_AWARE = datetime.datetime.fromtimestamp(0, utc)
EPOCH_NAIVE = EPOCH_AWARE.replace(tzinfo=None)
_DATETIME_ERROR_SUGGESTION = (
"(Consider Using CodecOptions(datetime_conversion=DATETIME_AUTO)"
" or MongoClient(datetime_conversion='DATETIME_AUTO'))."
" See: https://www.mongodb.com/docs/languages/python/pymongo-driver/current/data-formats/dates-and-times/#handling-out-of-range-datetimes"
)
class DatetimeMS:
"""Represents a BSON UTC datetime."""
__slots__ = ("_value",)
def __init__(self, value: Union[int, datetime.datetime]):
"""Represents a BSON UTC datetime.
BSON UTC datetimes are defined as an int64 of milliseconds since the
Unix epoch. The principal use of DatetimeMS is to represent
datetimes outside the range of the Python builtin
:class:`~datetime.datetime` class when
encoding/decoding BSON.
To decode UTC datetimes as a ``DatetimeMS``, `datetime_conversion` in
:class:`~bson.codec_options.CodecOptions` must be set to 'datetime_ms' or
'datetime_auto'. See `handling out of range datetimes <https://www.mongodb.com/docs/languages/python/pymongo-driver/current/data-formats/dates-and-times/#handling-out-of-range-datetimes>`_ for
details.
:param value: An instance of :class:`datetime.datetime` to be
represented as milliseconds since the Unix epoch, or int of
milliseconds since the Unix epoch.
"""
if isinstance(value, int):
if not (-(2**63) <= value <= 2**63 - 1):
raise OverflowError("Must be a 64-bit integer of milliseconds")
self._value = value
elif isinstance(value, datetime.datetime):
self._value = _datetime_to_millis(value)
else:
raise TypeError(f"{type(value)} is not a valid type for DatetimeMS")
def __hash__(self) -> int:
return hash(self._value)
def __repr__(self) -> str:
return type(self).__name__ + "(" + str(self._value) + ")"
def __lt__(self, other: Union[DatetimeMS, int]) -> bool:
return self._value < other
def __le__(self, other: Union[DatetimeMS, int]) -> bool:
return self._value <= other
def __eq__(self, other: Any) -> bool:
if isinstance(other, DatetimeMS):
return self._value == other._value
return False
def __ne__(self, other: Any) -> bool:
if isinstance(other, DatetimeMS):
return self._value != other._value
return True
def __gt__(self, other: Union[DatetimeMS, int]) -> bool:
return self._value > other
def __ge__(self, other: Union[DatetimeMS, int]) -> bool:
return self._value >= other
_type_marker = 9
def as_datetime(
self, codec_options: CodecOptions[Any] = DEFAULT_CODEC_OPTIONS
) -> datetime.datetime:
"""Create a Python :class:`~datetime.datetime` from this DatetimeMS object.
:param codec_options: A CodecOptions instance for specifying how the
resulting DatetimeMS object will be formatted using ``tz_aware``
and ``tz_info``. Defaults to
:const:`~bson.codec_options.DEFAULT_CODEC_OPTIONS`.
"""
return cast(datetime.datetime, _millis_to_datetime(self._value, codec_options))
def __int__(self) -> int:
return self._value
def _datetime_to_millis(dtm: datetime.datetime) -> int:
"""Convert datetime to milliseconds since epoch UTC."""
if dtm.utcoffset() is not None:
dtm = dtm - dtm.utcoffset() # type: ignore
return int(calendar.timegm(dtm.timetuple()) * 1000 + dtm.microsecond // 1000)
_MIN_UTC = datetime.datetime.min.replace(tzinfo=utc)
_MAX_UTC = datetime.datetime.max.replace(tzinfo=utc)
_MIN_UTC_MS = _datetime_to_millis(_MIN_UTC)
_MAX_UTC_MS = _datetime_to_millis(_MAX_UTC)
# Inclusive min and max for timezones.
def _min_datetime_ms(tz: datetime.tzinfo = utc) -> int:
delta = tz.utcoffset(_MIN_UTC)
if delta is not None:
offset_millis = (delta.days * 86400 + delta.seconds) * 1000 + delta.microseconds // 1000
else:
offset_millis = 0
return max(_MIN_UTC_MS, _MIN_UTC_MS - offset_millis)
def _max_datetime_ms(tz: datetime.tzinfo = utc) -> int:
delta = tz.utcoffset(_MAX_UTC)
if delta is not None:
offset_millis = (delta.days * 86400 + delta.seconds) * 1000 + delta.microseconds // 1000
else:
offset_millis = 0
return min(_MAX_UTC_MS, _MAX_UTC_MS - offset_millis)
def _millis_to_datetime(
millis: int, opts: CodecOptions[Any]
) -> Union[datetime.datetime, DatetimeMS]:
"""Convert milliseconds since epoch UTC to datetime."""
if (
opts.datetime_conversion == DatetimeConversion.DATETIME
or opts.datetime_conversion == DatetimeConversion.DATETIME_CLAMP
or opts.datetime_conversion == DatetimeConversion.DATETIME_AUTO
):
tz = opts.tzinfo or utc
if opts.datetime_conversion == DatetimeConversion.DATETIME_CLAMP:
millis = max(_min_datetime_ms(tz), min(millis, _max_datetime_ms(tz)))
elif opts.datetime_conversion == DatetimeConversion.DATETIME_AUTO:
if not (_min_datetime_ms(tz) <= millis <= _max_datetime_ms(tz)):
return DatetimeMS(millis)
diff = ((millis % 1000) + 1000) % 1000
seconds = (millis - diff) // 1000
micros = diff * 1000
try:
if opts.tz_aware:
dt = EPOCH_AWARE + datetime.timedelta(seconds=seconds, microseconds=micros)
if opts.tzinfo:
dt = dt.astimezone(tz)
return dt
else:
return EPOCH_NAIVE + datetime.timedelta(seconds=seconds, microseconds=micros)
except ArithmeticError as err:
raise InvalidBSON(f"{err} {_DATETIME_ERROR_SUGGESTION}") from err
elif opts.datetime_conversion == DatetimeConversion.DATETIME_MS:
return DatetimeMS(millis)
else:
raise ValueError("datetime_conversion must be an element of DatetimeConversion")

View File

@ -13,121 +13,135 @@
# limitations under the License. # limitations under the License.
"""Tools for manipulating DBRefs (references to MongoDB documents).""" """Tools for manipulating DBRefs (references to MongoDB documents)."""
from __future__ import annotations
from copy import deepcopy from copy import deepcopy
from typing import Any, Mapping, Optional
from bson._helpers import _getstate_slots, _setstate_slots
from bson.son import SON from bson.son import SON
class DBRef: class DBRef(object):
"""A reference to a document stored in MongoDB.""" """A reference to a document stored in MongoDB.
"""
__slots__ = "__collection", "__id", "__database", "__kwargs"
__getstate__ = _getstate_slots
__setstate__ = _setstate_slots
# DBRef isn't actually a BSON "type" so this number was arbitrarily chosen. # DBRef isn't actually a BSON "type" so this number was arbitrarily chosen.
_type_marker = 100 _type_marker = 100
def __init__( def __init__(self, collection, id, database=None, _extra={}, **kwargs):
self,
collection: str,
id: Any,
database: Optional[str] = None,
_extra: Optional[Mapping[str, Any]] = None,
**kwargs: Any,
) -> None:
"""Initialize a new :class:`DBRef`. """Initialize a new :class:`DBRef`.
Raises :class:`TypeError` if `collection` or `database` is not Raises :class:`TypeError` if `collection` or `database` is not
an instance of :class:`str`. `database` is optional and allows an instance of :class:`basestring` (:class:`str` in python 3).
references to documents to work across databases. Any additional `database` is optional and allows references to documents to work
keyword arguments will create additional fields in the resultant across databases. Any additional keyword arguments will create
embedded document. additional fields in the resultant embedded document.
:param collection: name of the collection the document is stored in :Parameters:
:param id: the value of the document's ``"_id"`` field - `collection`: name of the collection the document is stored in
:param database: name of the database to reference - `id`: the value of the document's ``"_id"`` field
:param kwargs: additional keyword arguments will - `database` (optional): name of the database to reference
- `**kwargs` (optional): additional keyword arguments will
create additional, custom fields create additional, custom fields
.. seealso:: The MongoDB documentation on `dbrefs <https://dochub.mongodb.org/core/dbrefs>`_. .. versionchanged:: 1.8
Now takes keyword arguments to specify additional fields.
.. versionadded:: 1.1.1
The `database` parameter.
.. mongodoc:: dbrefs
""" """
if not isinstance(collection, str): if not isinstance(collection, basestring):
raise TypeError(f"collection must be an instance of str, not {type(collection)}") raise TypeError("collection must be an "
if database is not None and not isinstance(database, str): "instance of %s" % (basestring.__name__,))
raise TypeError(f"database must be an instance of str, not {type(database)}") if database is not None and not isinstance(database, basestring):
raise TypeError("database must be an "
"instance of %s" % (basestring.__name__,))
self.__collection = collection self.__collection = collection
self.__id = id self.__id = id
self.__database = database self.__database = database
kwargs.update(_extra or {}) kwargs.update(_extra)
self.__kwargs = kwargs self.__kwargs = kwargs
@property @property
def collection(self) -> str: def collection(self):
"""Get the name of this DBRef's collection.""" """Get the name of this DBRef's collection as unicode.
"""
return self.__collection return self.__collection
@property @property
def id(self) -> Any: def id(self):
"""Get this DBRef's _id.""" """Get this DBRef's _id.
"""
return self.__id return self.__id
@property @property
def database(self) -> Optional[str]: def database(self):
"""Get the name of this DBRef's database. """Get the name of this DBRef's database.
Returns None if this DBRef doesn't specify a database. Returns None if this DBRef doesn't specify a database.
.. versionadded:: 1.1.1
""" """
return self.__database return self.__database
def __getattr__(self, key: Any) -> Any: def __getattr__(self, key):
try: try:
return self.__kwargs[key] return self.__kwargs[key]
except KeyError: except KeyError:
raise AttributeError(key) from None raise AttributeError(key)
def as_doc(self) -> SON[str, Any]: # Have to provide __setstate__ to avoid
# infinite recursion since we override
# __getattr__.
def __setstate__(self, state):
self.__dict__.update(state)
def as_doc(self):
"""Get the SON document representation of this DBRef. """Get the SON document representation of this DBRef.
Generally not needed by application developers Generally not needed by application developers
""" """
doc = SON([("$ref", self.collection), ("$id", self.id)]) doc = SON([("$ref", self.collection),
("$id", self.id)])
if self.database is not None: if self.database is not None:
doc["$db"] = self.database doc["$db"] = self.database
doc.update(self.__kwargs) doc.update(self.__kwargs)
return doc return doc
def __repr__(self) -> str: def __repr__(self):
extra = "".join([f", {k}={v!r}" for k, v in self.__kwargs.items()]) extra = "".join([", %s=%r" % (k, v)
for k, v in self.__kwargs.iteritems()])
if self.database is None: if self.database is None:
return f"DBRef({self.collection!r}, {self.id!r}{extra})" return "DBRef(%r, %r%s)" % (self.collection, self.id, extra)
return f"DBRef({self.collection!r}, {self.id!r}, {self.database!r}{extra})" return "DBRef(%r, %r, %r%s)" % (self.collection, self.id,
self.database, extra)
def __eq__(self, other: Any) -> bool: def __eq__(self, other):
if isinstance(other, DBRef): if isinstance(other, DBRef):
us = (self.__database, self.__collection, self.__id, self.__kwargs) us = (self.__database, self.__collection,
them = (other.__database, other.__collection, other.__id, other.__kwargs) self.__id, self.__kwargs)
them = (other.__database, other.__collection,
other.__id, other.__kwargs)
return us == them return us == them
return NotImplemented return NotImplemented
def __ne__(self, other: Any) -> bool: def __ne__(self, other):
return not self == other return not self == other
def __hash__(self) -> int: def __hash__(self):
"""Get a hash value for this :class:`DBRef`.""" """Get a hash value for this :class:`DBRef`.
return hash(
(self.__collection, self.__id, self.__database, tuple(sorted(self.__kwargs.items())))
)
def __deepcopy__(self, memo: Any) -> DBRef: .. versionadded:: 1.1
"""Support function for `copy.deepcopy()`.""" """
return DBRef( return hash((self.__collection, self.__id, self.__database,
deepcopy(self.__collection, memo), tuple(sorted(self.__kwargs.items()))))
deepcopy(self.__id, memo),
deepcopy(self.__database, memo), def __deepcopy__(self, memo):
deepcopy(self.__kwargs, memo), """Support function for `copy.deepcopy()`.
)
.. versionadded:: 1.10
"""
return DBRef(deepcopy(self.__collection, memo),
deepcopy(self.__id, memo),
deepcopy(self.__database, memo),
deepcopy(self.__kwargs, memo))

View File

@ -1,351 +0,0 @@
# Copyright 2016-present MongoDB, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Tools for working with the BSON decimal128 type.
.. versionadded:: 3.4
"""
from __future__ import annotations
import decimal
import struct
from decimal import Decimal
from typing import Any, Sequence, Tuple, Type, Union
from bson.codec_options import TypeDecoder, TypeEncoder
_PACK_64 = struct.Struct("<Q").pack
_UNPACK_64 = struct.Struct("<Q").unpack
_EXPONENT_MASK = 3 << 61
_EXPONENT_BIAS = 6176
_EXPONENT_MAX = 6144
_EXPONENT_MIN = -6143
_MAX_DIGITS = 34
_INF = 0x7800000000000000
_NAN = 0x7C00000000000000
_SNAN = 0x7E00000000000000
_SIGN = 0x8000000000000000
_NINF = (_INF + _SIGN, 0)
_PINF = (_INF, 0)
_NNAN = (_NAN + _SIGN, 0)
_PNAN = (_NAN, 0)
_NSNAN = (_SNAN + _SIGN, 0)
_PSNAN = (_SNAN, 0)
_CTX_OPTIONS = {
"prec": _MAX_DIGITS,
"rounding": decimal.ROUND_HALF_EVEN,
"Emin": _EXPONENT_MIN,
"Emax": _EXPONENT_MAX,
"capitals": 1,
"flags": [],
"traps": [decimal.InvalidOperation, decimal.Overflow, decimal.Inexact],
"clamp": 1,
}
_DEC128_CTX = decimal.Context(**_CTX_OPTIONS.copy()) # type: ignore
_VALUE_OPTIONS = Union[decimal.Decimal, float, str, Tuple[int, Sequence[int], int]]
class DecimalEncoder(TypeEncoder):
"""Converts Python :class:`decimal.Decimal` to BSON :class:`Decimal128`.
For example::
opts = CodecOptions(type_registry=TypeRegistry([DecimalEncoder()]))
bson.encode({"d": decimal.Decimal('1.0')}, codec_options=opts)
.. versionadded:: 4.15
"""
@property
def python_type(self) -> Type[Decimal]:
return Decimal
def transform_python(self, value: Any) -> Decimal128:
return Decimal128(value)
class DecimalDecoder(TypeDecoder):
"""Converts BSON :class:`Decimal128` to Python :class:`decimal.Decimal`.
For example::
opts = CodecOptions(type_registry=TypeRegistry([DecimalDecoder()]))
bson.decode(data, codec_options=opts)
.. versionadded:: 4.15
"""
@property
def bson_type(self) -> Type[Decimal128]:
return Decimal128
def transform_bson(self, value: Any) -> decimal.Decimal:
return value.to_decimal()
def create_decimal128_context() -> decimal.Context:
"""Returns an instance of :class:`decimal.Context` appropriate
for working with IEEE-754 128-bit decimal floating point values.
"""
opts = _CTX_OPTIONS.copy()
opts["traps"] = []
return decimal.Context(**opts) # type: ignore
def _decimal_to_128(value: _VALUE_OPTIONS) -> Tuple[int, int]:
"""Converts a decimal.Decimal to BID (high bits, low bits).
:param value: An instance of decimal.Decimal
"""
with decimal.localcontext(_DEC128_CTX) as ctx:
value = ctx.create_decimal(value)
if value.is_infinite():
return _NINF if value.is_signed() else _PINF
sign, digits, exponent = value.as_tuple()
if value.is_nan():
if digits:
raise ValueError("NaN with debug payload is not supported")
if value.is_snan():
return _NSNAN if value.is_signed() else _PSNAN
return _NNAN if value.is_signed() else _PNAN
significand = int("".join([str(digit) for digit in digits]))
bit_length = significand.bit_length()
high = 0
low = 0
for i in range(min(64, bit_length)):
if significand & (1 << i):
low |= 1 << i
for i in range(64, bit_length):
if significand & (1 << i):
high |= 1 << (i - 64)
biased_exponent = exponent + _EXPONENT_BIAS # type: ignore[operator]
if high >> 49 == 1:
high = high & 0x7FFFFFFFFFFF
high |= _EXPONENT_MASK
high |= (biased_exponent & 0x3FFF) << 47
else:
high |= biased_exponent << 49
if sign:
high |= _SIGN
return high, low
class Decimal128:
"""BSON Decimal128 type::
>>> Decimal128(Decimal("0.0005"))
Decimal128('0.0005')
>>> Decimal128("0.0005")
Decimal128('0.0005')
>>> Decimal128((3474527112516337664, 5))
Decimal128('0.0005')
:param value: An instance of :class:`decimal.Decimal`, string, or tuple of
(high bits, low bits) from Binary Integer Decimal (BID) format.
.. note:: :class:`~Decimal128` uses an instance of :class:`decimal.Context`
configured for IEEE-754 Decimal128 when validating parameters.
Signals like :class:`decimal.InvalidOperation`, :class:`decimal.Inexact`,
and :class:`decimal.Overflow` are trapped and raised as exceptions::
>>> Decimal128(".13.1")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
...
decimal.InvalidOperation: [<class 'decimal.ConversionSyntax'>]
>>>
>>> Decimal128("1E-6177")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
...
decimal.Inexact: [<class 'decimal.Inexact'>]
>>>
>>> Decimal128("1E6145")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
...
decimal.Overflow: [<class 'decimal.Overflow'>, <class 'decimal.Rounded'>]
To ensure the result of a calculation can always be stored as BSON
Decimal128 use the context returned by
:func:`create_decimal128_context`::
>>> import decimal
>>> decimal128_ctx = create_decimal128_context()
>>> with decimal.localcontext(decimal128_ctx) as ctx:
... Decimal128(ctx.create_decimal(".13.3"))
...
Decimal128('NaN')
>>>
>>> with decimal.localcontext(decimal128_ctx) as ctx:
... Decimal128(ctx.create_decimal("1E-6177"))
...
Decimal128('0E-6176')
>>>
>>> with decimal.localcontext(DECIMAL128_CTX) as ctx:
... Decimal128(ctx.create_decimal("1E6145"))
...
Decimal128('Infinity')
To match the behavior of MongoDB's Decimal128 implementation
str(Decimal(value)) may not match str(Decimal128(value)) for NaN values::
>>> Decimal128(Decimal('NaN'))
Decimal128('NaN')
>>> Decimal128(Decimal('-NaN'))
Decimal128('NaN')
>>> Decimal128(Decimal('sNaN'))
Decimal128('NaN')
>>> Decimal128(Decimal('-sNaN'))
Decimal128('NaN')
However, :meth:`~Decimal128.to_decimal` will return the exact value::
>>> Decimal128(Decimal('NaN')).to_decimal()
Decimal('NaN')
>>> Decimal128(Decimal('-NaN')).to_decimal()
Decimal('-NaN')
>>> Decimal128(Decimal('sNaN')).to_decimal()
Decimal('sNaN')
>>> Decimal128(Decimal('-sNaN')).to_decimal()
Decimal('-sNaN')
Two instances of :class:`Decimal128` compare equal if their Binary
Integer Decimal encodings are equal::
>>> Decimal128('NaN') == Decimal128('NaN')
True
>>> Decimal128('NaN').bid == Decimal128('NaN').bid
True
This differs from :class:`decimal.Decimal` comparisons for NaN::
>>> Decimal('NaN') == Decimal('NaN')
False
"""
__slots__ = ("__high", "__low")
_type_marker = 19
def __init__(self, value: _VALUE_OPTIONS) -> None:
if isinstance(value, (str, decimal.Decimal)):
self.__high, self.__low = _decimal_to_128(value)
elif isinstance(value, (list, tuple)):
if len(value) != 2:
raise ValueError(
"Invalid size for creation of Decimal128 "
"from list or tuple. Must have exactly 2 "
"elements."
)
self.__high, self.__low = value
else:
raise TypeError(f"Cannot convert {value!r} to Decimal128")
def to_decimal(self) -> decimal.Decimal:
"""Returns an instance of :class:`decimal.Decimal` for this
:class:`Decimal128`.
"""
high = self.__high
low = self.__low
sign = 1 if (high & _SIGN) else 0
if (high & _SNAN) == _SNAN:
return decimal.Decimal((sign, (), "N")) # type: ignore
elif (high & _NAN) == _NAN:
return decimal.Decimal((sign, (), "n")) # type: ignore
elif (high & _INF) == _INF:
return decimal.Decimal((sign, (), "F")) # type: ignore
if (high & _EXPONENT_MASK) == _EXPONENT_MASK:
exponent = ((high & 0x1FFFE00000000000) >> 47) - _EXPONENT_BIAS
return decimal.Decimal((sign, (0,), exponent))
else:
exponent = ((high & 0x7FFF800000000000) >> 49) - _EXPONENT_BIAS
arr = bytearray(15)
mask = 0x00000000000000FF
for i in range(14, 6, -1):
arr[i] = (low & mask) >> ((14 - i) << 3)
mask = mask << 8
mask = 0x00000000000000FF
for i in range(6, 0, -1):
arr[i] = (high & mask) >> ((6 - i) << 3)
mask = mask << 8
mask = 0x0001000000000000
arr[0] = (high & mask) >> 48
# cdecimal only accepts a tuple for digits.
digits = tuple(int(digit) for digit in str(int.from_bytes(arr, "big")))
with decimal.localcontext(_DEC128_CTX) as ctx:
return ctx.create_decimal((sign, digits, exponent))
@classmethod
def from_bid(cls: Type[Decimal128], value: bytes) -> Decimal128:
"""Create an instance of :class:`Decimal128` from Binary Integer
Decimal string.
:param value: 16 byte string (128-bit IEEE 754-2008 decimal floating
point in Binary Integer Decimal (BID) format).
"""
if not isinstance(value, bytes):
raise TypeError(f"value must be an instance of bytes, not {type(value)}")
if len(value) != 16:
raise ValueError("value must be exactly 16 bytes")
return cls((_UNPACK_64(value[8:])[0], _UNPACK_64(value[:8])[0])) # type: ignore
@property
def bid(self) -> bytes:
"""The Binary Integer Decimal (BID) encoding of this instance."""
return _PACK_64(self.__low) + _PACK_64(self.__high)
def __str__(self) -> str:
dec = self.to_decimal()
if dec.is_nan():
# Required by the drivers spec to match MongoDB behavior.
return "NaN"
return str(dec)
def __repr__(self) -> str:
return f"Decimal128('{self!s}')"
def __setstate__(self, value: Tuple[int, int]) -> None:
self.__high, self.__low = value
def __getstate__(self) -> Tuple[int, int]:
return self.__high, self.__low
def __eq__(self, other: Any) -> bool:
if isinstance(other, Decimal128):
return self.bid == other.bid
return NotImplemented
def __ne__(self, other: Any) -> bool:
return not self == other

118
bson/encoding_helpers.c Normal file
View File

@ -0,0 +1,118 @@
/*
* Copyright 2009-2015 MongoDB, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
#include "encoding_helpers.h"
/*
* Portions Copyright 2001 Unicode, Inc.
*
* Disclaimer
*
* This source code is provided as is by Unicode, Inc. No claims are
* made as to fitness for any particular purpose. No warranties of any
* kind are expressed or implied. The recipient agrees to determine
* applicability of information provided. If this file has been
* purchased on magnetic or optical media from Unicode, Inc., the
* sole remedy for any claim will be exchange of defective media
* within 90 days of receipt.
*
* Limitations on Rights to Redistribute This Code
*
* Unicode, Inc. hereby grants the right to freely use the information
* supplied in this file in the creation of products supporting the
* Unicode Standard, and to make copies of this file in any form
* for internal or external distribution as long as this notice
* remains attached.
*/
/*
* Index into the table below with the first byte of a UTF-8 sequence to
* get the number of trailing bytes that are supposed to follow it.
*/
static const char trailingBytesForUTF8[256] = {
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,
2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2, 3,3,3,3,3,3,3,3,4,4,4,4,5,5,5,5
};
/* --------------------------------------------------------------------- */
/*
* Utility routine to tell whether a sequence of bytes is legal UTF-8.
* This must be called with the length pre-determined by the first byte.
* The length can be set by:
* length = trailingBytesForUTF8[*source]+1;
* and the sequence is illegal right away if there aren't that many bytes
* available.
* If presented with a length > 4, this returns 0. The Unicode
* definition of UTF-8 goes up to 4-byte sequences.
*/
static unsigned char isLegalUTF8(const unsigned char* source, int length) {
unsigned char a;
const unsigned char* srcptr = source + length;
switch (length) {
default: return 0;
/* Everything else falls through when "true"... */
case 4: if ((a = (*--srcptr)) < 0x80 || a > 0xBF) return 0;
case 3: if ((a = (*--srcptr)) < 0x80 || a > 0xBF) return 0;
case 2: if ((a = (*--srcptr)) > 0xBF) return 0;
switch (*source) {
/* no fall-through in this inner switch */
case 0xE0: if (a < 0xA0) return 0; break;
case 0xF0: if (a < 0x90) return 0; break;
case 0xF4: if (a > 0x8F) return 0; break;
default: if (a < 0x80) return 0;
}
case 1: if (*source >= 0x80 && *source < 0xC2) return 0;
if (*source > 0xF4) return 0;
}
return 1;
}
result_t check_string(const unsigned char* string, const int length,
const char check_utf8, const char check_null) {
int position = 0;
/* By default we go character by character. Will be different for checking
* UTF-8 */
int sequence_length = 1;
if (!check_utf8 && !check_null) {
return VALID;
}
while (position < length) {
if (check_null && *(string + position) == 0) {
return HAS_NULL;
}
if (check_utf8) {
sequence_length = trailingBytesForUTF8[*(string + position)] + 1;
if ((position + sequence_length) > length) {
return NOT_UTF_8;
}
if (!isLegalUTF8(string + position, sequence_length)) {
return NOT_UTF_8;
}
}
position += sequence_length;
}
return VALID;
}

29
bson/encoding_helpers.h Normal file
View File

@ -0,0 +1,29 @@
/*
* Copyright 2009-2015 MongoDB, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
#ifndef ENCODING_HELPERS_H
#define ENCODING_HELPERS_H
typedef enum {
VALID,
NOT_UTF_8,
HAS_NULL
} result_t;
result_t check_string(const unsigned char* string, const int length,
const char check_utf8, const char check_null);
#endif

View File

@ -1,4 +1,4 @@
# Copyright 2009-present MongoDB, Inc. # Copyright 2009-2015 MongoDB, Inc.
# #
# Licensed under the Apache License, Version 2.0 (the "License"); # Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License. # you may not use this file except in compliance with the License.
@ -13,37 +13,28 @@
# limitations under the License. # limitations under the License.
"""Exceptions raised by the BSON package.""" """Exceptions raised by the BSON package."""
from __future__ import annotations
from typing import Any, Optional
class BSONError(Exception): class BSONError(Exception):
"""Base class for all BSON exceptions.""" """Base class for all BSON exceptions.
"""
class InvalidBSON(BSONError): class InvalidBSON(BSONError):
"""Raised when trying to create a BSON object from invalid data.""" """Raised when trying to create a BSON object from invalid data.
"""
class InvalidStringData(BSONError): class InvalidStringData(BSONError):
"""Raised when trying to encode a string containing non-UTF8 data.""" """Raised when trying to encode a string containing non-UTF8 data.
"""
class InvalidDocument(BSONError): class InvalidDocument(BSONError):
"""Raised when trying to create a BSON object from an invalid document.""" """Raised when trying to create a BSON object from an invalid document.
"""
def __init__(self, message: str, document: Optional[Any] = None) -> None:
super().__init__(message)
self._document = document
@property
def document(self) -> Any:
"""The invalid document that caused the error.
..versionadded:: 4.16"""
return self._document
class InvalidId(BSONError): class InvalidId(BSONError):
"""Raised when trying to create an ObjectId from invalid data.""" """Raised when trying to create an ObjectId from invalid data.
"""

View File

@ -1,39 +0,0 @@
# Copyright 2014-2015 MongoDB, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""A BSON wrapper for long (int in python3)"""
from __future__ import annotations
from typing import Any
class Int64(int):
"""Representation of the BSON int64 type.
This is necessary because every integral number is an :class:`int` in
Python 3. Small integral numbers are encoded to BSON int32 by default,
but Int64 numbers will always be encoded to BSON int64.
:param value: the numeric value to represent
"""
__slots__ = ()
_type_marker = 18
def __getstate__(self) -> Any:
return {}
def __setstate__(self, state: Any) -> None:
pass

File diff suppressed because it is too large Load Diff

Some files were not shown because too many files have changed in this diff Show More