This PR adds an option to specify file to job mapping explicitly during
task creation. This option is incompatible with most other job-related
parameters like `sorting_method` and `frame_step`.
- Added a new task creation parameter (`job_file_mapping`) to set a
custom file to job mapping during task creation
Django REST Framework ignores the Content-Type on request body parts, so
it doesn't know that they are JSON-encoded. Instead, it just tries to
decode each part as if it was an `str()`-encoded value.
Change the encoding to match the decoding. The only type this matters
for is `str`, because `json.dumps` and `str` produce different encodings
for `str` values.
Remove `none_type` from the list of encodable types since, to my
knowledge, there's no way to encode a `None` value as a
`multipart/form-data` part in a way that DRF will understand.
PR removes previews downloading from CS when task creating (skipped in
PR #5478). In addition, I had to change the test to check for the file
name existing in the message when the specified file is not found in the
bucket, because now the preview is no longer downloaded at the stage of
creating a task.
This will let users to run their PyTorch code without network access,
provided that they have already cached the data.
### How has this been tested?
<!-- Please describe in detail how you tested your changes.
Include details of your testing environment, and the tests you ran to
see how your change affects other areas of the code, etc. -->
Unit tests.
Turns out that #5456 had a nasty side effect. Session hooks are called
when pytest is run with `--collect-only` (even though no tests are
actually run in this case), and Visual Studio Code periodically runs
`pytest --collect-only` in order to learn what tests exist in the
project. As a result, it keeps restarting the services and restoring the
database in the background.
Work around this by skipping all logic in the hooks if `--collect-only`
is in the options.
Extracted from https://github.com/opencv/cvat/pull/5083
- Added a default arg for task data uploading
- Added an option to wait for the data processing in task data uploading
- Moved data splitting by requests for TUS closer to the point of use
This PR fixes a problem with local running of tests introduced in #5498.
When running tests locally, there can be multiple tags available, while
we need the `dev` tag, which should be created right before tests are
executed.
- Added the missing image tag in the docker run command
It's possible to specify only the manifest file and filename pattern for
creating task with cloud storage data.
The special characters supported now for the pattern are `*`, `?`,
`[seq]`, `[!seq]`.
Please see
[here](8898a8b264/tests/python/rest_api/test_tasks.py (L686))
for some examples of how to use this functionality.
Co-authored-by: Maxim Zhiltsov <zhiltsov.max35@gmail.com>
Related: #5225
External services are not available when we execute in Helm.
- Added a way to mark REST API tests that require external services to
run
- Changed the way of filtering tests in Helm tests
Currently, we can't execute external service mocks in tests, and we
ignore related tests in the Helm
execution command. But this command may be outdated, because Helm tests
are not executed in each PR.
This PR allows to indicate related tests and filter them out without the
need to synchronize the CI command.
OPA can take some time to load rules, but our tests don't wait for OPA,
and start right after the server is loaded.
Sometimes it works, but in other times the tests may fail because OPA is
still loading the rules.
This PR allows to wait for OPA during the test suite startup.
This should protect us from unnoticed hangs, which happen sometimes and
waste CI and developer time.
Currently, the limit is 15s per test, which should be enough in normal
conditions.
Extracted some enhancements from
https://github.com/opencv/cvat/pull/4819
- Extracted common s3 manipulations in tests
- Refactored import/export tests to be more clear
For user-facing functions, keep accepting `str` paths to maintain
compatibility and flexibility, but add support for arbitrary path-like
objects. For internal functions (in `downloading.py` and
`uploading.py`), don't bother and require `pathlib.Path`.
The only code that isn't converted is build-time code (e.g. `setup.py`)
and code that came from openapi-generator.
`pathlib` improves code readability and type safety. It is already used
in some of the tests; convert all remaining `os.path` usage to `pathlib`
equivalents.
Fixed REST API tests after merging #5408 and #5396 to develop
Co-authored-by: Boris Sekachev <sekachev.bs@gmail.com>
Co-authored-by: Nikita Manovich <nikita@cvat.ai>
* Fix TUS offset queries in production environments
Previously, `mod_wsgi` would convert `HEAD` requests into `GET`, which
would be rejected, so clients were unable to resume an upload that failed
midway through.
To make use of this, update the SDK code to enable upload resumption.
* Add a test that forces a chunked TUS upload
* test_can_create_from_backup_in_chunks: make sure the upload is actually chunked
* add tests for try of creating multitasks with images and video
* add tests seccess creating video with Remote source
* reduce time of debounce
* try fix 107, 118 tests
* try fix 107 tests
* fix test with attach video
* disable test with attach video
* fix url to remote videos
* Aborted extra changes
* Updated files
* Refactored case 107
* Fixed almost all tests
* Aborted extra changes
* removed extra files
* Improved tests, added tests for local videos
* Fixed corner case
* Fix ://
* Redesigned a way of getting a file type
* Added html type to button
* Some refactoring
* Removed extra file
* sdk tests: fix path for file share images
Co-authored-by: Boris <sekachev.bs@gmail.com>
Co-authored-by: kirill-sizov <sizow.k.d@gmail.com>
* Move the test file share contents to tests/share
This way, it can be reused between tests.
* cvat-sdk: Fix creating tasks with non-local files
Forcing the `Content-Type` for the `upload_data` API call to
`multipart/form-data` does not work, because the current logic for
converting Python values to parts (`ApiClient._convert_body_to_post_params`)
does not encode them in a way that Django REST Framework can understand (it
JSON-encodes each part).
Fortunately, we don't actually need to do that, since when we create a task
with non-local files, we don't need to upload any files, and so we can just
post the original JSON, so do just that.
I couldn't add a test for the remote image case, because CVAT rejects all URLs
with non-public IP addressses. However, I did test this case manually.
Maxim Zhiltsov
Maria Khrustaleva
Kirill Sizov
Roman Donchenko
Andrey Zhavoronkov
Anastasia Yasakova
Nikita Manovich
dependabot[bot]
magician
Boris Sekachev
Aleksey Alekseev