Compare commits

...

38 Commits

Author SHA1 Message Date
527285bb13 Remove redundant line to create edit_form 2025-01-20 12:31:38 +01:00
19fa308c06 Update chill bundles to version 3.6.0 2025-01-16 18:00:48 +01:00
1b831bc424 Fix activity between dates filter: condition added for alias 2025-01-16 15:19:38 +01:00
573118e514 Undo change migration 2025-01-16 13:05:18 +01:00
0cabf5654a Add changie for fix 2025-01-16 12:25:42 +01:00
cfb547d55f Increase length of varchar for id chill_person_marital_status 2025-01-16 12:23:59 +01:00
a915c35026 Rector changes 2025-01-16 10:25:02 +01:00
018f8aef5c Add export button to social actions template 2025-01-15 16:45:05 +01:00
de6385ba21 Refactor SocialIssuesExportController to include method for exporting social actions 2025-01-15 16:44:45 +01:00
edb51dd3cd Add export button to template 2025-01-15 16:35:09 +01:00
c379bccad4 Create social issue export controller 2025-01-15 16:33:57 +01:00
bd9ad8a569 remove prettier command from yarn 2025-01-15 13:08:26 +01:00
cb5fd2b69d Merge branch 'address-importer-ban' into 'master'
Add service and command to import French addresses from BAN

See merge request Chill-Projet/chill-bundles!781
2025-01-13 16:09:17 +00:00
feebcf6662 Add changie [ci-skip] 2025-01-13 17:08:45 +01:00
6de4861b98 Fix email attachment handling in address import reports
Updated attachment logic to use in-memory file contents and apply a `.gz` suffix to filenames. This ensures better file handling and resolves potential issues with attaching files directly from a path.
2025-01-10 23:00:40 +01:00
b4a1e824ac Add service and command to import French addresses from BAN
Introduce a service to handle the import of French addresses from the Base Adresse Nationale (BAN) dataset. Add a new console command `chill:main:address-ref-from-ban` to trigger the import by department numbers, with an option to send a report email for unmatched addresses.
2025-01-10 22:52:08 +01:00
d87cf925e2 Add changie file for storing document on disk feature 2025-01-09 16:28:36 +01:00
ce3cce7b95 Merge branch '346-store-docs-on-disk' into 'master'
Resolve "Permettre de stocker les documents sur disque, localement."

Closes #346

See merge request Chill-Projet/chill-bundles!774
2025-01-09 15:16:45 +00:00
6c97654e5e Add documentation for document storage configuration
Introduce a new guide detailing document storage options, including on-disk storage and cloud-based OpenStack integration. This document explains configuration steps, benefits, and limitations for both methods, and is now linked in the production installation index.
2025-01-09 16:05:58 +01:00
0787e61c22 Set default configuration file for chill_doc_store 2025-01-09 15:25:44 +01:00
73bcfb82b7 Add configuration option to select storage driver
Introduces a new `use_driver` configuration option to specify the desired storage driver (`local_storage` or `openstack`). Ensures proper validation to handle multiple drivers and throws appropriate errors when configurations are inconsistent or missing. Refactors related logic to improve clarity and maintainability.
2025-01-09 15:25:43 +01:00
812e4047d0 Adjust key size in KeyGenerator to 32 bytes.
Changed the key size from 128 bytes to 32 bytes in the KeyGenerator service. This aligns with the expected algorithm requirements and ensures proper cryptographic behavior.
2025-01-09 15:25:43 +01:00
999ac3af2b Add TempUrl signature validation to local storage
Implemented local storage-based file handling with TempUrl signature validation for upload and retrieval. Added validation checks for parameters like max file size/count, expiration, and signature integrity. Included unit tests for TempUrl signature validation and adjusted configuration for local storage.
2025-01-09 15:25:42 +01:00
0c628c39db store encrypted content 2025-01-09 15:25:42 +01:00
c65f1d495d Refactor ConfigureOpenstackObjectStorageCommand
- change namespace for more obvious handling;
- remove command of local storage is configured
2025-01-09 15:25:41 +01:00
83f7086bb0 Configure DI for providing kernel secret for TempUrlLocalStorageGenerator 2025-01-09 15:25:41 +01:00
c1e449f48e Implements StoredObjectManager for local storage 2025-01-09 15:25:41 +01:00
1f6de3cb11 Implement TempUrlLocalStorageGenerator and its tests
Added the full implementation for TempUrlLocalStorageGenerator, including methods to generate signed URLs and POST requests with expiration and signature logic. Introduced corresponding unit tests to validate functionality using mocked dependencies.
2025-01-09 15:25:40 +01:00
3a2548ed89 Select storage depending on configuration 2025-01-09 15:25:39 +01:00
d7652658f2 Define new configuration for local storage 2025-01-09 15:25:39 +01:00
67b5bc6dba Implements required interface to store documents on disk 2025-01-09 15:25:38 +01:00
e25c1e1816 Refactor object storage to separate local storage and openstack storage 2025-01-09 15:25:38 +01:00
282b7f7fbb Merge branch 'import-addresses-handle-no-postcode' into 'master'
Allow addresses without postal code to be imported without failure, and add email reporting for unimported addresses in import commands

See merge request Chill-Projet/chill-bundles!780
2025-01-09 11:52:21 +00:00
ab311eaecb Add email reporting for unimported addresses in import commands
Enhanced address import commands to optionally send a recap of unimported addresses via email. Updated import logic to handle cases where postal codes are missing, log issues, and generate compressed CSV reports with failed entries.
2025-01-09 12:21:10 +01:00
edcc01149b Merge branch 'master' of gitlab.com:Chill-Projet/chill-bundles 2025-01-07 16:59:32 +01:00
27b2d77fdb Fix EntityToJsonTransformer for saved exports 2025-01-07 16:59:16 +01:00
96bb98f854 upgrade yarn deps 2025-01-07 10:03:40 +01:00
68ed2db51e Refactor exception type to InvalidConfigurationException.
Replaced InvalidArgumentException with InvalidConfigurationException for widget service alias conflicts. This ensures the exception better reflects the configuration-related nature of the error.
2025-01-07 10:03:24 +01:00
50 changed files with 2454 additions and 662 deletions

3
.changes/v3.5.3.md Normal file
View File

@@ -0,0 +1,3 @@
## v3.5.3 - 2025-01-07
### Fixed
* Fix the EntityToJsonTransformer to return an empty array if the value is ""

9
.changes/v3.6.0.md Normal file
View File

@@ -0,0 +1,9 @@
## v3.6.0 - 2025-01-16
### Feature
* Importer for addresses does not fails when the postal code is not found with some addresses, and compute a recap list of all addresses that could not be imported. This recap list can be send by email.
* ([#346](https://gitlab.com/Chill-Projet/chill-bundles/-/issues/346)) Create a driver for storing documents on disk (instead of openstack object store)
* Add address importer from french Base d'Adresse Nationale (BAN)
* ([#343](https://gitlab.com/Chill-Projet/chill-bundles/-/issues/343)) Add csv export for social issues and social actions
### Fixed
* Export: fix missing alias in activity between certain dates filter. Condition added for alias.

View File

@@ -6,6 +6,20 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html),
and is generated by [Changie](https://github.com/miniscruff/changie).
## v3.6.0 - 2025-01-16
### Feature
* Importer for addresses does not fails when the postal code is not found with some addresses, and compute a recap list of all addresses that could not be imported. This recap list can be send by email.
* ([#346](https://gitlab.com/Chill-Projet/chill-bundles/-/issues/346)) Create a driver for storing documents on disk (instead of openstack object store)
* Add address importer from french Base d'Adresse Nationale (BAN)
* ([#343](https://gitlab.com/Chill-Projet/chill-bundles/-/issues/343)) Add csv export for social issues and social actions
### Fixed
* Export: fix missing alias in activity between certain dates filter. Condition added for alias.
## v3.5.3 - 2025-01-07
### Fixed
* Fix the EntityToJsonTransformer to return an empty array if the value is ""
## v3.5.2 - 2024-12-19
### Fixed
* ([#345](https://gitlab.com/Chill-Projet/chill-bundles/-/issues/345)) Export: activity filtering of users that were associated to an activity between certain dates. Results contained activities that were not within the specified date range"

View File

@@ -13,6 +13,7 @@
"ext-json": "*",
"ext-openssl": "*",
"ext-redis": "*",
"ext-zlib": "*",
"champs-libres/wopi-bundle": "dev-master@dev",
"champs-libres/wopi-lib": "dev-master@dev",
"doctrine/doctrine-bundle": "^2.1",

View File

@@ -1,4 +1,7 @@
chill_doc_store:
use_driver: openstack
local_storage:
storage_path: '%kernel.project_dir%/var/storage'
openstack:
temp_url:
temp_url_key: '%env(resolve:ASYNC_UPLOAD_TEMP_URL_KEY)%' # Required

View File

@@ -0,0 +1,84 @@
Document storage
################
You can store document on two different ways:
- on disk
- in the cloud, using object storage: currently only `openstack swift <https://docs.openstack.org/api-ref/object-store/index.html>`_ is supported.
Comparison
==========
Storing documents within the cloud is particularily suitable for "portable" deployments, like in kubernetes, or within container
without having to manage volumes to store documents. But you'll have to subscribe on a commercial offer.
Storing documents on disk is more easy to configure, but more difficult to manage: if you use container, you will have to
manager volumes to attach documents on disk. You'll have to do some backup of the directory. If chill is load-balanced (and
multiple instances of chill are run), you will have to find a way to share the directories in read-write mode for every instance.
On Disk
=======
Configure Chill like this:
.. code-block:: yaml
# file config/packages/chill_doc_store.yaml
chill_doc_store:
use_driver: local_storage
local_storage:
storage_path: '%kernel.project_dir%/var/storage'
In this configuration, documents will be stored in :code:`var/storage` within your app directory. But this path can be
elsewhere on the disk. Be aware that the directory must be writable by the user executing the chill app (php-fpm or www-data).
Documents will be stored in subpathes within that directory. The files will be encrypted, the key is stored in the database.
In the cloud, using openstack object store
##########################################
You must subscribe to a commercial offer for object store.
Chill use some features to allow documents to be stored in the cloud without being uploaded first to the chill server:
- `Form POST Middelware <https://docs.openstack.org/swift/latest/api/form_post_middleware.html>`_;
- `Temporary URL Middelware <https://docs.openstack.org/swift/latest/api/temporary_url_middleware.html>`_.
A secret key must be generated and configured, and CORS must be configured depending on the domain you will use to serve Chill.
At first, create a container and get the base path to the container. For instance, on OVH, if you create a container named "mychill",
you will be able to retrieve the base path of the container within the OVH interface, like this:
- base_path: :code:`https://storage.gra.cloud.ovh.net/v1/AUTH_123456789/mychill/` => will be variable :code:`ASYNC_UPLOAD_TEMP_URL_BASE_PATH`
- container: :code:`mychill` => will be variable :code:`ASYNC_UPLOAD_TEMP_URL_CONTAINER`
You can also generate a key, which should have at least 20 characters. This key will go in the variable :code:`ASYNC_UPLOAD_TEMP_URL_KEY`.
.. note::
See the `documentation of symfony <https://symfony.com/doc/current/configuration.html#config-env-vars>`_ on how to store variables, and how to encrypt them if needed.
Configure the storage like this:
.. code-block:: yaml
# file config/packages/chill_doc_store.yaml
chill_doc_store:
use_driver: openstack
openstack:
temp_url:
temp_url_key: '%env(resolve:ASYNC_UPLOAD_TEMP_URL_KEY)%' # Required
container: '%env(resolve:ASYNC_UPLOAD_TEMP_URL_CONTAINER)%' # Required
temp_url_base_path: '%env(resolve:ASYNC_UPLOAD_TEMP_URL_BASE_PATH)%' # Required
Chill is able to configure the container in order to store document. Grab an Openstack Token (for instance, using :code:`openstack token issue` or
the web interface of your openstack provider), and run this command:
.. code-block:: bash
symfony console async-upload:configure --os_token=OPENSTACK_TOKEN -d https://mychill.mydomain.example
# or, without symfony-cli
bin/console async-upload:configure --os_token=OPENSTACK_TOKEN -d https://mychill.mydomain.example

View File

@@ -323,6 +323,7 @@ Going further
:maxdepth: 2
prod.rst
document-storage.rst
load-addresses.rst
prod-calendar-sms-sending.rst
msgraph-configure.rst

View File

@@ -78,7 +78,6 @@
"scripts": {
"dev-server": "encore dev-server",
"dev": "encore dev",
"prettier": "prettier --write \"**/*.{js,ts,vue}\"",
"watch": "encore dev --watch",
"build": "encore production --progress",
"eslint": "npx eslint-baseline \"**/*.{js,ts,vue}\""

View File

@@ -55,7 +55,9 @@ final readonly class PersonHavingActivityBetweenDateFilter implements ExportElem
.' AND '
.'(person_person_having_activity.id = person.id OR person MEMBER OF activity_person_having_activity.persons)');
$sqb->andWhere('activity_person_having_activity.id = activity.id');
if (\in_array('activity', $qb->getAllAliases(), true)) {
$sqb->andWhere('activity_person_having_activity.id = activity.id');
}
if (isset($data['reasons']) && [] !== $data['reasons']) {
// add clause activity reason

View File

@@ -0,0 +1,227 @@
<?php
declare(strict_types=1);
/*
* Chill is a software for social workers
*
* For the full copyright and license information, please view
* the LICENSE file that was distributed with this source code.
*/
namespace Chill\DocStoreBundle\AsyncUpload\Driver\LocalStorage;
use Base64Url\Base64Url;
use Chill\DocStoreBundle\Entity\StoredObject;
use Chill\DocStoreBundle\Entity\StoredObjectVersion;
use Chill\DocStoreBundle\Exception\StoredObjectManagerException;
use Chill\DocStoreBundle\Service\Cryptography\KeyGenerator;
use Chill\DocStoreBundle\Service\StoredObjectManagerInterface;
use Symfony\Component\DependencyInjection\ParameterBag\ParameterBagInterface;
use Symfony\Component\Filesystem\Filesystem;
use Symfony\Component\Filesystem\Path;
class StoredObjectManager implements StoredObjectManagerInterface
{
private readonly string $baseDir;
private readonly Filesystem $filesystem;
public function __construct(
ParameterBagInterface $parameterBag,
private readonly KeyGenerator $keyGenerator,
) {
$this->baseDir = $parameterBag->get('chill_doc_store')['local_storage']['storage_path'];
$this->filesystem = new Filesystem();
}
public function getLastModified(StoredObject|StoredObjectVersion $document): \DateTimeInterface
{
$version = $document instanceof StoredObject ? $document->getCurrentVersion() : $document;
if (null === $version) {
throw StoredObjectManagerException::storedObjectDoesNotContainsVersion();
}
$path = $this->buildPath($version->getFilename());
if (false === $ts = filemtime($path)) {
throw StoredObjectManagerException::unableToReadDocumentOnDisk($path);
}
return \DateTimeImmutable::createFromFormat('U', (string) $ts);
}
public function getContentLength(StoredObject|StoredObjectVersion $document): int
{
return strlen($this->read($document));
}
public function exists(StoredObject|StoredObjectVersion $document): bool
{
$version = $document instanceof StoredObject ? $document->getCurrentVersion() : $document;
if (null === $version) {
return false;
}
return $this->existsContent($version->getFilename());
}
public function read(StoredObject|StoredObjectVersion $document): string
{
$version = $document instanceof StoredObject ? $document->getCurrentVersion() : $document;
if (null === $version) {
throw StoredObjectManagerException::storedObjectDoesNotContainsVersion();
}
$content = $this->readContent($version->getFilename());
if (!$this->isVersionEncrypted($version)) {
return $content;
}
$clearData = openssl_decrypt(
$content,
self::ALGORITHM,
// TODO: Why using this library and not use base64_decode() ?
Base64Url::decode($version->getKeyInfos()['k']),
\OPENSSL_RAW_DATA,
pack('C*', ...$version->getIv())
);
if (false === $clearData) {
throw StoredObjectManagerException::unableToDecrypt(openssl_error_string());
}
return $clearData;
}
public function write(StoredObject $document, string $clearContent, ?string $contentType = null): StoredObjectVersion
{
$newIv = $document->isEncrypted() ? $document->getIv() : $this->keyGenerator->generateIv();
$newKey = $document->isEncrypted() ? $document->getKeyInfos() : $this->keyGenerator->generateKey(self::ALGORITHM);
$newType = $contentType ?? $document->getType();
$version = $document->registerVersion(
$newIv,
$newKey,
$newType
);
$encryptedContent = $this->isVersionEncrypted($version)
? openssl_encrypt(
$clearContent,
self::ALGORITHM,
// TODO: Why using this library and not use base64_decode() ?
Base64Url::decode($version->getKeyInfos()['k']),
\OPENSSL_RAW_DATA,
pack('C*', ...$version->getIv())
)
: $clearContent;
if (false === $encryptedContent) {
throw StoredObjectManagerException::unableToEncryptDocument((string) openssl_error_string());
}
$this->writeContent($version->getFilename(), $encryptedContent);
return $version;
}
public function readContent(string $filename): string
{
$path = $this->buildPath($filename);
if (!file_exists($path)) {
throw StoredObjectManagerException::unableToFindDocumentOnDisk($path);
}
if (false === $content = file_get_contents($path)) {
throw StoredObjectManagerException::unableToReadDocumentOnDisk($path);
}
return $content;
}
public function writeContent(string $filename, string $encryptedContent): void
{
$fullPath = $this->buildPath($filename);
$dir = Path::getDirectory($fullPath);
if (!$this->filesystem->exists($dir)) {
$this->filesystem->mkdir($dir);
}
$result = file_put_contents($fullPath, $encryptedContent);
if (false === $result) {
throw StoredObjectManagerException::unableToStoreDocumentOnDisk();
}
}
public function existsContent(string $filename): bool
{
$path = $this->buildPath($filename);
return $this->filesystem->exists($path);
}
private function buildPath(string $filename): string
{
$dirs = [$this->baseDir];
for ($i = 0; $i < min(strlen($filename), 8); ++$i) {
$dirs[] = $filename[$i];
}
$dirs[] = $filename;
return Path::canonicalize(implode(DIRECTORY_SEPARATOR, $dirs));
}
public function delete(StoredObjectVersion $storedObjectVersion): void
{
if (!$this->exists($storedObjectVersion)) {
return;
}
$path = $this->buildPath($storedObjectVersion->getFilename());
$this->filesystem->remove($path);
$this->removeDirectoriesRecursively(Path::getDirectory($path));
}
private function removeDirectoriesRecursively(string $path): void
{
if ($path === $this->baseDir) {
return;
}
$files = scandir($path);
// if it does contains only "." and "..", we can remove the directory
if (2 === count($files) && in_array('.', $files, true) && in_array('..', $files, true)) {
$this->filesystem->remove($path);
$this->removeDirectoriesRecursively(Path::getDirectory($path));
}
}
/**
* @throws StoredObjectManagerException
*/
public function etag(StoredObject|StoredObjectVersion $document): string
{
return md5($this->read($document));
}
public function clearCache(): void
{
// there is no cache: nothing to do here !
}
private function isVersionEncrypted(StoredObjectVersion $storedObjectVersion): bool
{
return $storedObjectVersion->isEncrypted();
}
}

View File

@@ -0,0 +1,107 @@
<?php
declare(strict_types=1);
/*
* Chill is a software for social workers
*
* For the full copyright and license information, please view
* the LICENSE file that was distributed with this source code.
*/
namespace Chill\DocStoreBundle\AsyncUpload\Driver\LocalStorage;
use Chill\DocStoreBundle\AsyncUpload\SignedUrl;
use Chill\DocStoreBundle\AsyncUpload\SignedUrlPost;
use Chill\DocStoreBundle\AsyncUpload\TempUrlGeneratorInterface;
use Chill\DocStoreBundle\Entity\StoredObject;
use Symfony\Component\Clock\ClockInterface;
use Symfony\Component\Routing\Generator\UrlGeneratorInterface;
class TempUrlLocalStorageGenerator implements TempUrlGeneratorInterface
{
private const SIGNATURE_DURATION = 180;
public function __construct(
private readonly string $secret,
private readonly ClockInterface $clock,
private readonly UrlGeneratorInterface $urlGenerator,
) {}
public function generate(string $method, string $object_name, ?int $expire_delay = null): SignedUrl
{
$expiration = $this->clock->now()->getTimestamp() + min($expire_delay ?? self::SIGNATURE_DURATION, self::SIGNATURE_DURATION);
return new SignedUrl(
strtoupper($method),
$this->urlGenerator->generate('chill_docstore_stored_object_operate', [
'object_name' => $object_name,
'exp' => $expiration,
'sig' => $this->sign(strtoupper($method), $object_name, $expiration),
], UrlGeneratorInterface::ABSOLUTE_URL),
\DateTimeImmutable::createFromFormat('U', (string) $expiration),
$object_name,
);
}
public function generatePost(?int $expire_delay = null, ?int $submit_delay = null, int $max_file_count = 1, ?string $object_name = null): SignedUrlPost
{
$submitDelayComputed = min($submit_delay ?? self::SIGNATURE_DURATION, self::SIGNATURE_DURATION);
$expireDelayComputed = min($expire_delay ?? self::SIGNATURE_DURATION, self::SIGNATURE_DURATION);
$objectNameComputed = $object_name ?? StoredObject::generatePrefix();
$expiration = $this->clock->now()->getTimestamp() + $expireDelayComputed + $submitDelayComputed;
return new SignedUrlPost(
$this->urlGenerator->generate(
'chill_docstore_storedobject_post',
['prefix' => $objectNameComputed],
UrlGeneratorInterface::ABSOLUTE_URL
),
\DateTimeImmutable::createFromFormat('U', (string) $expiration),
$objectNameComputed,
15_000_000,
1,
$submitDelayComputed,
'',
$objectNameComputed,
$this->sign('POST', $object_name, $expiration),
);
}
private function sign(string $method, string $object_name, int $expiration): string
{
return hash('sha512', sprintf('%s.%s.%s.%d', $method, $this->secret, $object_name, $expiration));
}
public function validateSignaturePost(string $signature, string $prefix, int $expiration, int $maxFileSize, int $maxFileCount): bool
{
if (15_000_000 !== $maxFileSize || 1 !== $maxFileCount) {
return false;
}
return $this->internalValidateSignature($signature, 'POST', $prefix, $expiration);
}
private function internalValidateSignature(string $signature, string $method, string $object_name, int $expiration): bool
{
if ($expiration < $this->clock->now()->format('U')) {
return false;
}
if ('' === $object_name) {
return false;
}
return $this->sign($method, $object_name, $expiration) === $signature;
}
public function validateSignature(string $signature, string $method, string $objectName, int $expiration): bool
{
if (!in_array($method, ['GET', 'HEAD'], true)) {
return false;
}
return $this->internalValidateSignature($signature, $method, $objectName, $expiration);
}
}

View File

@@ -9,7 +9,7 @@ declare(strict_types=1);
* the LICENSE file that was distributed with this source code.
*/
namespace Chill\DocStoreBundle\AsyncUpload\Command;
namespace Chill\DocStoreBundle\AsyncUpload\Driver\OpenstackObjectStore;
use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Input\InputInterface;

View File

@@ -9,13 +9,14 @@ declare(strict_types=1);
* the LICENSE file that was distributed with this source code.
*/
namespace Chill\DocStoreBundle\Service;
namespace Chill\DocStoreBundle\AsyncUpload\Driver\OpenstackObjectStore;
use Base64Url\Base64Url;
use Chill\DocStoreBundle\AsyncUpload\TempUrlGeneratorInterface;
use Chill\DocStoreBundle\Entity\StoredObject;
use Chill\DocStoreBundle\Entity\StoredObjectVersion;
use Chill\DocStoreBundle\Exception\StoredObjectManagerException;
use Chill\DocStoreBundle\Service\StoredObjectManagerInterface;
use Symfony\Component\HttpFoundation\Request;
use Symfony\Component\HttpFoundation\Response;
use Symfony\Contracts\HttpClient\Exception\TransportExceptionInterface;
@@ -24,8 +25,6 @@ use Symfony\Contracts\HttpClient\ResponseInterface;
final class StoredObjectManager implements StoredObjectManagerInterface
{
private const ALGORITHM = 'AES-256-CBC';
private array $inMemory = [];
public function __construct(
@@ -361,6 +360,6 @@ final class StoredObjectManager implements StoredObjectManagerInterface
private function isVersionEncrypted(StoredObjectVersion $storedObjectVersion): bool
{
return ([] !== $storedObjectVersion->getKeyInfos()) && ([] !== $storedObjectVersion->getIv());
return $storedObjectVersion->isEncrypted();
}
}

View File

@@ -11,6 +11,7 @@ declare(strict_types=1);
namespace Chill\DocStoreBundle;
use Chill\DocStoreBundle\DependencyInjection\Compiler\StorageConfigurationCompilerPass;
use Chill\DocStoreBundle\GenericDoc\GenericDocForAccompanyingPeriodProviderInterface;
use Chill\DocStoreBundle\GenericDoc\GenericDocForPersonProviderInterface;
use Chill\DocStoreBundle\GenericDoc\Twig\GenericDocRendererInterface;
@@ -27,5 +28,7 @@ class ChillDocStoreBundle extends Bundle
->addTag('chill_doc_store.generic_doc_person_provider');
$container->registerForAutoconfiguration(GenericDocRendererInterface::class)
->addTag('chill_doc_store.generic_doc_renderer');
$container->addCompilerPass(new StorageConfigurationCompilerPass());
}
}

View File

@@ -0,0 +1,120 @@
<?php
declare(strict_types=1);
/*
* Chill is a software for social workers
*
* For the full copyright and license information, please view
* the LICENSE file that was distributed with this source code.
*/
namespace Chill\DocStoreBundle\Controller;
use Chill\DocStoreBundle\AsyncUpload\Driver\LocalStorage\StoredObjectManager;
use Chill\DocStoreBundle\AsyncUpload\Driver\LocalStorage\TempUrlLocalStorageGenerator;
use Symfony\Component\HttpFoundation\File\UploadedFile;
use Symfony\Component\HttpFoundation\Request;
use Symfony\Component\HttpFoundation\Response;
use Symfony\Component\HttpKernel\Exception\AccessDeniedHttpException;
use Symfony\Component\HttpKernel\Exception\BadRequestHttpException;
use Symfony\Component\HttpKernel\Exception\NotFoundHttpException;
use Symfony\Component\Routing\Annotation\Route;
/**
* Controller to deal with local storage operation.
*/
final readonly class StoredObjectContentToLocalStorageController
{
public function __construct(
private StoredObjectManager $storedObjectManager,
private TempUrlLocalStorageGenerator $tempUrlLocalStorageGenerator,
) {}
#[Route('/public/stored-object/post', name: 'chill_docstore_storedobject_post', methods: ['POST'])]
public function postContent(Request $request): Response
{
$prefix = $request->query->get('prefix', '');
if ('' === $prefix) {
throw new BadRequestHttpException('Prefix parameter is missing');
}
if (0 === $maxFileSize = $request->request->getInt('max_file_size', 0)) {
throw new BadRequestHttpException('Max file size is not set or equal to zero');
}
if (1 !== $maxFileCount = $request->request->getInt('max_file_count', 0)) {
throw new BadRequestHttpException('Max file count is not set or equal to zero');
}
if (0 === $expiration = $request->request->getInt('expires', 0)) {
throw new BadRequestHttpException('Expiration is not set or equal to zero');
}
if ('' === $signature = $request->request->get('signature', '')) {
throw new BadRequestHttpException('Signature is not set or is a blank string');
}
if (!$this->tempUrlLocalStorageGenerator->validateSignaturePost($signature, $prefix, $expiration, $maxFileSize, $maxFileCount)) {
throw new AccessDeniedHttpException('Invalid signature');
}
$keyFiles = $request->files->keys();
if ($maxFileCount < count($keyFiles)) {
throw new AccessDeniedHttpException('More files than max file count');
}
if (0 === count($keyFiles)) {
throw new BadRequestHttpException('Zero files given');
}
foreach ($keyFiles as $keyFile) {
/** @var UploadedFile $file */
$file = $request->files->get($keyFile);
if ($maxFileSize < strlen($file->getContent())) {
throw new AccessDeniedHttpException('File is too big');
}
if (!str_starts_with((string) $keyFile, $prefix)) {
throw new AccessDeniedHttpException('Filename does not start with signed prefix');
}
$this->storedObjectManager->writeContent($keyFile, $file->getContent());
}
return new Response(status: Response::HTTP_NO_CONTENT);
}
#[Route('/public/stored-object/operate', name: 'chill_docstore_stored_object_operate', methods: ['GET', 'HEAD'])]
public function contentOperate(Request $request): Response
{
if ('' === $objectName = $request->query->get('object_name', '')) {
throw new BadRequestHttpException('Object name parameter is missing');
}
if (0 === $expiration = $request->query->getInt('exp', 0)) {
throw new BadRequestHttpException('Expiration is not set or equal to zero');
}
if ('' === $signature = $request->query->get('sig', '')) {
throw new BadRequestHttpException('Signature is not set or is a blank string');
}
if (!$this->tempUrlLocalStorageGenerator->validateSignature($signature, strtoupper($request->getMethod()), $objectName, $expiration)) {
throw new AccessDeniedHttpException('Invalid signature');
}
if (!$this->storedObjectManager->existsContent($objectName)) {
throw new NotFoundHttpException('Object does not exists on disk');
}
return match ($request->getMethod()) {
'GET' => new Response($this->storedObjectManager->readContent($objectName)),
'HEAD' => new Response(''),
default => throw new BadRequestHttpException('method not supported'),
};
}
}

View File

@@ -53,7 +53,7 @@ class ChillDocStoreExtension extends Extension implements PrependExtensionInterf
$this->prependTwig($container);
}
protected function prependAuthorization(ContainerBuilder $container)
private function prependAuthorization(ContainerBuilder $container)
{
$container->prependExtensionConfig('security', [
'role_hierarchy' => [
@@ -69,7 +69,7 @@ class ChillDocStoreExtension extends Extension implements PrependExtensionInterf
]);
}
protected function prependRoute(ContainerBuilder $container)
private function prependRoute(ContainerBuilder $container)
{
// declare routes for task bundle
$container->prependExtensionConfig('chill_main', [
@@ -81,7 +81,7 @@ class ChillDocStoreExtension extends Extension implements PrependExtensionInterf
]);
}
protected function prependTwig(ContainerBuilder $container)
private function prependTwig(ContainerBuilder $container)
{
$twigConfig = [
'form_themes' => ['@ChillDocStore/Form/fields.html.twig'],

View File

@@ -0,0 +1,86 @@
<?php
declare(strict_types=1);
/*
* Chill is a software for social workers
*
* For the full copyright and license information, please view
* the LICENSE file that was distributed with this source code.
*/
namespace Chill\DocStoreBundle\DependencyInjection\Compiler;
use Chill\DocStoreBundle\AsyncUpload\Driver\LocalStorage\TempUrlLocalStorageGenerator;
use Chill\DocStoreBundle\AsyncUpload\Driver\OpenstackObjectStore\ConfigureOpenstackObjectStorageCommand;
use Chill\DocStoreBundle\AsyncUpload\Driver\OpenstackObjectStore\TempUrlOpenstackGenerator;
use Chill\DocStoreBundle\AsyncUpload\TempUrlGeneratorInterface;
use Chill\DocStoreBundle\Controller\StoredObjectContentToLocalStorageController;
use Chill\DocStoreBundle\Service\StoredObjectManagerInterface;
use Symfony\Component\Config\Definition\Exception\InvalidConfigurationException;
use Symfony\Component\DependencyInjection\Compiler\CompilerPassInterface;
use Symfony\Component\DependencyInjection\ContainerBuilder;
class StorageConfigurationCompilerPass implements CompilerPassInterface
{
private const SERVICES_OPENSTACK = [
\Chill\DocStoreBundle\AsyncUpload\Driver\OpenstackObjectStore\StoredObjectManager::class,
TempUrlOpenstackGenerator::class,
ConfigureOpenstackObjectStorageCommand::class,
];
private const SERVICES_LOCAL_STORAGE = [
\Chill\DocStoreBundle\AsyncUpload\Driver\LocalStorage\StoredObjectManager::class,
TempUrlLocalStorageGenerator::class,
StoredObjectContentToLocalStorageController::class,
];
public function process(ContainerBuilder $container)
{
$config = $container
->getParameterBag()
->resolveValue($container->getParameter('chill_doc_store'));
if (array_key_exists('local_storage', $config) && !array_key_exists('openstack', $config)) {
$driver = 'local_storage';
$this->checkUseDriverConfiguration($config['use_driver'] ?? null, $driver);
} elseif (!array_key_exists('local_storage', $config) && array_key_exists('openstack', $config)) {
$driver = 'openstack';
$this->checkUseDriverConfiguration($config['use_driver'] ?? null, $driver);
} elseif (array_key_exists('openstack', $config) && array_key_exists('local_storage', $config)) {
$driver = $config['use_driver'] ?? null;
if (null === $driver) {
throw new InvalidConfigurationException('There are multiple drivers configured for chill_doc_store, set the one you want to use with the variable use_driver');
}
} else {
throw new InvalidConfigurationException('No driver defined for storing document. Define one in chill_doc_store configuration');
}
if ('local_storage' === $driver) {
foreach (self::SERVICES_OPENSTACK as $service) {
$container->removeDefinition($service);
}
$container->setAlias(StoredObjectManagerInterface::class, \Chill\DocStoreBundle\AsyncUpload\Driver\LocalStorage\StoredObjectManager::class);
$container->setAlias(TempUrlGeneratorInterface::class, TempUrlLocalStorageGenerator::class);
} else {
foreach (self::SERVICES_LOCAL_STORAGE as $service) {
$container->removeDefinition($service);
}
$container->setAlias(StoredObjectManagerInterface::class, \Chill\DocStoreBundle\AsyncUpload\Driver\OpenstackObjectStore\StoredObjectManager::class);
$container->setAlias(TempUrlGeneratorInterface::class, TempUrlOpenstackGenerator::class);
}
}
private function checkUseDriverConfiguration(?string $useDriver, string $driver): void
{
if (null === $useDriver) {
return;
}
if ($useDriver !== $driver) {
throw new InvalidConfigurationException(sprintf('The "use_driver" configuration require a driver (%s) which is not configured. Configure this driver in order to use it.', $useDriver));
}
}
}

View File

@@ -30,10 +30,22 @@ class Configuration implements ConfigurationInterface
/* @phpstan-ignore-next-line As there are inconsistencies in return types, but the code works... */
$rootNode->children()
->enumNode('use_driver')
->values(['local_storage', 'openstack'])
->info('Driver to use. Default to the single one if multiple driver are defined. Configuration will raise an error if there are multiple drivers defined, and if this key is not set')
->end()
->arrayNode('local_storage')
->info('where the stored object should be stored')
->children()
->scalarNode('storage_path')
->info('the folder where the stored object should be stored')
->isRequired()->cannotBeEmpty()
->end() // end of storage_path
->end() // end of children
->end() // end of local_storage
// openstack node
->arrayNode('openstack')
->info('parameters to authenticate and generate temp url against the openstack object storage service')
->addDefaultsIfNotSet()
->children()
// openstack.temp_url
->arrayNode('temp_url')

View File

@@ -448,4 +448,12 @@ class StoredObject implements Document, TrackCreationInterface
{
return $storedObject->getDeleteAt() < $now && $storedObject->getVersions()->isEmpty();
}
/**
* Return true if it has a current version, and if the current version is encrypted.
*/
public function isEncrypted(): bool
{
return $this->hasCurrentVersion() && $this->getCurrentVersion()->isEncrypted();
}
}

View File

@@ -226,4 +226,9 @@ class StoredObjectVersion implements TrackCreationInterface
return $this;
}
public function isEncrypted(): bool
{
return ([] !== $this->getKeyInfos()) && ([] !== $this->getIv());
}
}

View File

@@ -34,4 +34,29 @@ final class StoredObjectManagerException extends \Exception
{
return new self('Unable to get content from response.', 500, $exception);
}
public static function unableToStoreDocumentOnDisk(?\Throwable $exception = null): self
{
return new self('Unable to store document on disk.', previous: $exception);
}
public static function unableToFindDocumentOnDisk(string $path): self
{
return new self('Unable to find document on disk at path "'.$path.'".');
}
public static function unableToReadDocumentOnDisk(string $path): self
{
return new self('Unable to read document on disk at path "'.$path.'".');
}
public static function unableToEncryptDocument(string $errors): self
{
return new self('Unable to encrypt document: '.$errors);
}
public static function storedObjectDoesNotContainsVersion(): self
{
return new self('Stored object does not contains any version');
}
}

View File

@@ -0,0 +1,59 @@
<?php
declare(strict_types=1);
/*
* Chill is a software for social workers
*
* For the full copyright and license information, please view
* the LICENSE file that was distributed with this source code.
*/
namespace Chill\DocStoreBundle\Service\Cryptography;
use Base64Url\Base64Url;
use Chill\DocStoreBundle\Service\StoredObjectManagerInterface;
use Random\Randomizer;
class KeyGenerator
{
private readonly Randomizer $randomizer;
public function __construct()
{
$this->randomizer = new Randomizer();
}
/**
* @return array{alg: string, ext: bool, k: string, key_ops: list<string>, kty: string}
*/
public function generateKey(string $algo = StoredObjectManagerInterface::ALGORITHM): array
{
if (StoredObjectManagerInterface::ALGORITHM !== $algo) {
throw new \LogicException(sprintf("Algorithm '%s' is not supported.", $algo));
}
$key = $this->randomizer->getBytes(32);
return [
'alg' => 'A256CBC',
'ext' => true,
'k' => Base64Url::encode($key),
'key_ops' => ['encrypt', 'decrypt'],
'kty' => 'oct',
];
}
/**
* @return list<int<0, 255>>
*/
public function generateIv(): array
{
$iv = [];
for ($i = 0; $i < 16; ++$i) {
$iv[] = unpack('C', $this->randomizer->getBytes(8))[1];
}
return $iv;
}
}

View File

@@ -18,6 +18,8 @@ use Symfony\Contracts\HttpClient\Exception\TransportExceptionInterface;
interface StoredObjectManagerInterface
{
public const ALGORITHM = 'AES-256-CBC';
/**
* @param StoredObject|StoredObjectVersion $document if a StoredObject is given, the last version will be used
*/

View File

@@ -0,0 +1,160 @@
<?php
declare(strict_types=1);
/*
* Chill is a software for social workers
*
* For the full copyright and license information, please view
* the LICENSE file that was distributed with this source code.
*/
namespace Chill\DocStoreBundle\Tests\AsyncUpload\Driver\LocalStorage;
use Chill\DocStoreBundle\AsyncUpload\Driver\LocalStorage\StoredObjectManager;
use Chill\DocStoreBundle\Entity\StoredObject;
use Chill\DocStoreBundle\Entity\StoredObjectVersion;
use Chill\DocStoreBundle\Service\Cryptography\KeyGenerator;
use PHPUnit\Framework\TestCase;
use Symfony\Component\DependencyInjection\ParameterBag\ParameterBag;
/**
* @internal
*
* @coversNothing
*/
class StoredObjectManagerTest extends TestCase
{
private const CONTENT = 'abcde';
public function testWrite(): StoredObjectVersion
{
$storedObject = new StoredObject();
$manager = $this->buildStoredObjectManager();
$version = $manager->write($storedObject, self::CONTENT);
self::assertSame($storedObject, $version->getStoredObject());
return $version;
}
/**
* @depends testWrite
*/
public function testRead(StoredObjectVersion $version): StoredObjectVersion
{
$manager = $this->buildStoredObjectManager();
$content = $manager->read($version);
self::assertEquals(self::CONTENT, $content);
return $version;
}
/**
* @depends testRead
*/
public function testExists(StoredObjectVersion $version): StoredObjectVersion
{
$manager = $this->buildStoredObjectManager();
$notExisting = new StoredObject();
$versionNotPersisted = $notExisting->registerVersion();
self::assertTrue($manager->exists($version));
self::assertFalse($manager->exists($versionNotPersisted));
self::assertFalse($manager->exists(new StoredObject()));
return $version;
}
/**
* @throws \Chill\DocStoreBundle\Exception\StoredObjectManagerException
*
* @depends testExists
*/
public function testEtag(StoredObjectVersion $version): StoredObjectVersion
{
$manager = $this->buildStoredObjectManager();
$actual = $manager->etag($version);
self::assertEquals(md5(self::CONTENT), $actual);
return $version;
}
/**
* @depends testEtag
*/
public function testGetContentLength(StoredObjectVersion $version): StoredObjectVersion
{
$manager = $this->buildStoredObjectManager();
$actual = $manager->getContentLength($version);
self::assertSame(5, $actual);
return $version;
}
/**
* @throws \Chill\DocStoreBundle\Exception\StoredObjectManagerException
*
* @depends testGetContentLength
*/
public function testGetLastModified(StoredObjectVersion $version): StoredObjectVersion
{
$manager = $this->buildStoredObjectManager();
$actual = $manager->getLastModified($version);
self::assertInstanceOf(\DateTimeImmutable::class, $actual);
self::assertGreaterThan((new \DateTimeImmutable('now'))->getTimestamp() - 10, $actual->getTimestamp());
return $version;
}
/**
* @depends testGetLastModified
*/
public function testDelete(StoredObjectVersion $version): void
{
$manager = $this->buildStoredObjectManager();
$manager->delete($version);
self::assertFalse($manager->exists($version));
}
public function testDeleteDoesNotRemoveOlderVersion(): void
{
$storedObject = new StoredObject();
$manager = $this->buildStoredObjectManager();
$version1 = $manager->write($storedObject, 'version1');
$version2 = $manager->write($storedObject, 'version2');
$version3 = $manager->write($storedObject, 'version3');
self::assertTrue($manager->exists($version1));
self::assertEquals('version1', $manager->read($version1));
self::assertTrue($manager->exists($version2));
self::assertEquals('version2', $manager->read($version2));
self::assertTrue($manager->exists($version3));
self::assertEquals('version3', $manager->read($version3));
// we delete the intermediate version
$manager->delete($version2);
self::assertFalse($manager->exists($version2));
// we check that we are still able to download the other versions
self::assertTrue($manager->exists($version1));
self::assertEquals('version1', $manager->read($version1));
self::assertTrue($manager->exists($version3));
self::assertEquals('version3', $manager->read($version3));
}
private function buildStoredObjectManager(): StoredObjectManager
{
return new StoredObjectManager(
new ParameterBag(['chill_doc_store' => ['local_storage' => ['storage_path' => '/tmp/chill-local-storage-test']]]),
new KeyGenerator(),
);
}
}

View File

@@ -0,0 +1,238 @@
<?php
declare(strict_types=1);
/*
* Chill is a software for social workers
*
* For the full copyright and license information, please view
* the LICENSE file that was distributed with this source code.
*/
namespace Chill\DocStoreBundle\Tests\AsyncUpload\Driver\LocalStorage;
use Chill\DocStoreBundle\AsyncUpload\Driver\LocalStorage\TempUrlLocalStorageGenerator;
use PHPUnit\Framework\TestCase;
use Prophecy\PhpUnit\ProphecyTrait;
use Symfony\Component\Clock\ClockInterface;
use Symfony\Component\Clock\MockClock;
use Symfony\Component\Routing\Generator\UrlGeneratorInterface;
/**
* @internal
*
* @coversNothing
*/
class TempUrlLocalStorageGeneratorTest extends TestCase
{
use ProphecyTrait;
private const SECRET = 'abc';
public function testGenerate(): void
{
$urlGenerator = $this->prophesize(UrlGeneratorInterface::class);
$urlGenerator->generate('chill_docstore_stored_object_operate', [
'object_name' => $object_name = 'testABC',
'exp' => $expiration = 1734307200 + 180,
'sig' => TempUrlLocalStorageGeneratorTest::expectedSignature('GET', $object_name, $expiration),
], UrlGeneratorInterface::ABSOLUTE_URL)
->shouldBeCalled()
->willReturn($url = 'http://example.com/public/doc-store/stored-object/operate/testABC');
$generator = $this->buildGenerator($urlGenerator->reveal());
$signedUrl = $generator->generate('GET', $object_name);
self::assertEquals($url, $signedUrl->url);
self::assertEquals($object_name, $signedUrl->object_name);
self::assertEquals($expiration, $signedUrl->expires->getTimestamp());
self::assertEquals('GET', $signedUrl->method);
}
public function testGeneratePost(): void
{
$urlGenerator = $this->prophesize(UrlGeneratorInterface::class);
$urlGenerator->generate('chill_docstore_storedobject_post', [
'prefix' => 'prefixABC',
], UrlGeneratorInterface::ABSOLUTE_URL)
->shouldBeCalled()
->willReturn($url = 'http://example.com/public/doc-store/stored-object/prefixABC');
$generator = $this->buildGenerator($urlGenerator->reveal());
$signedUrl = $generator->generatePost(object_name: 'prefixABC');
self::assertEquals($url, $signedUrl->url);
self::assertEquals('prefixABC', $signedUrl->object_name);
self::assertEquals($expiration = 1734307200 + 180 + 180, $signedUrl->expires->getTimestamp());
self::assertEquals('POST', $signedUrl->method);
self::assertEquals(TempUrlLocalStorageGeneratorTest::expectedSignature('POST', 'prefixABC', $expiration), $signedUrl->signature);
}
private static function expectedSignature(string $method, $objectName, int $expiration): string
{
return hash('sha512', sprintf('%s.%s.%s.%d', $method, self::SECRET, $objectName, $expiration));
}
/**
* @dataProvider generateValidateSignatureData
*/
public function testValidateSignature(string $signature, string $method, string $objectName, int $expiration, \DateTimeImmutable $now, bool $expected, string $message): void
{
$urlGenerator = $this->buildGenerator(clock: new MockClock($now));
self::assertEquals($expected, $urlGenerator->validateSignature($signature, $method, $objectName, $expiration), $message);
}
/**
* @dataProvider generateValidateSignaturePostData
*/
public function testValidateSignaturePost(string $signature, int $expiration, string $objectName, int $maxFileSize, int $maxFileCount, \DateTimeImmutable $now, bool $expected, string $message): void
{
$urlGenerator = $this->buildGenerator(clock: new MockClock($now));
self::assertEquals($expected, $urlGenerator->validateSignaturePost($signature, $objectName, $expiration, $maxFileSize, $maxFileCount), $message);
}
public static function generateValidateSignaturePostData(): iterable
{
yield [
TempUrlLocalStorageGeneratorTest::expectedSignature('POST', $object_name = 'testABC', $expiration = 1734307200 + 180),
$expiration,
$object_name,
15_000_000,
1,
\DateTimeImmutable::createFromFormat('U', (string) ($expiration - 10)),
true,
'Valid signature',
];
yield [
TempUrlLocalStorageGeneratorTest::expectedSignature('POST', $object_name = 'testABC', $expiration = 1734307200 + 180),
$expiration,
$object_name,
15_000_001,
1,
\DateTimeImmutable::createFromFormat('U', (string) ($expiration - 10)),
false,
'Wrong max file size',
];
yield [
TempUrlLocalStorageGeneratorTest::expectedSignature('POST', $object_name = 'testABC', $expiration = 1734307200 + 180),
$expiration,
$object_name,
15_000_000,
2,
\DateTimeImmutable::createFromFormat('U', (string) ($expiration - 10)),
false,
'Wrong max file count',
];
yield [
TempUrlLocalStorageGeneratorTest::expectedSignature('POST', $object_name = 'testABC', $expiration = 1734307200 + 180),
$expiration,
$object_name.'AAA',
15_000_000,
1,
\DateTimeImmutable::createFromFormat('U', (string) ($expiration - 10)),
false,
'Invalid object name',
];
yield [
TempUrlLocalStorageGeneratorTest::expectedSignature('POST', $object_name = 'testABC', $expiration = 1734307200 + 180).'A',
$expiration,
$object_name,
15_000_000,
1,
\DateTimeImmutable::createFromFormat('U', (string) ($expiration - 10)),
false,
'Invalid signature',
];
yield [
TempUrlLocalStorageGeneratorTest::expectedSignature('POST', $object_name = 'testABC', $expiration = 1734307200 + 180),
$expiration,
$object_name,
15_000_000,
1,
\DateTimeImmutable::createFromFormat('U', (string) ($expiration + 1)),
false,
'Expired signature',
];
}
public static function generateValidateSignatureData(): iterable
{
yield [
TempUrlLocalStorageGeneratorTest::expectedSignature('GET', $object_name = 'testABC', $expiration = 1734307200 + 180),
'GET',
$object_name,
$expiration,
\DateTimeImmutable::createFromFormat('U', (string) ($expiration - 10)),
true,
'Valid signature, not expired',
];
yield [
TempUrlLocalStorageGeneratorTest::expectedSignature('HEAD', $object_name = 'testABC', $expiration = 1734307200 + 180),
'HEAD',
$object_name,
$expiration,
\DateTimeImmutable::createFromFormat('U', (string) ($expiration - 10)),
true,
'Valid signature, not expired',
];
yield [
TempUrlLocalStorageGeneratorTest::expectedSignature('GET', $object_name = 'testABC', $expiration = 1734307200 + 180).'A',
'GET',
$object_name,
$expiration,
\DateTimeImmutable::createFromFormat('U', (string) ($expiration - 10)),
false,
'Invalid signature',
];
yield [
TempUrlLocalStorageGeneratorTest::expectedSignature('GET', $object_name = 'testABC', $expiration = 1734307200 + 180),
'GET',
$object_name,
$expiration,
\DateTimeImmutable::createFromFormat('U', (string) ($expiration + 1)),
false,
'Signature expired',
];
yield [
TempUrlLocalStorageGeneratorTest::expectedSignature('GET', $object_name = 'testABC', $expiration = 1734307200 + 180),
'GET',
$object_name.'____',
$expiration,
\DateTimeImmutable::createFromFormat('U', (string) ($expiration - 10)),
false,
'Invalid object name',
];
yield [
TempUrlLocalStorageGeneratorTest::expectedSignature('POST', $object_name = 'testABC', $expiration = 1734307200 + 180),
'POST',
$object_name,
$expiration,
\DateTimeImmutable::createFromFormat('U', (string) ($expiration - 10)),
false,
'Wrong method',
];
}
private function buildGenerator(?UrlGeneratorInterface $urlGenerator = null, ?ClockInterface $clock = null): TempUrlLocalStorageGenerator
{
return new TempUrlLocalStorageGenerator(
self::SECRET,
$clock ?? new MockClock('2024-12-16T00:00:00+00:00'),
$urlGenerator ?? $this->prophesize(UrlGeneratorInterface::class)->reveal(),
);
}
}

View File

@@ -9,9 +9,9 @@ declare(strict_types=1);
* the LICENSE file that was distributed with this source code.
*/
namespace AsyncUpload\Command;
namespace Chill\DocStoreBundle\Tests\AsyncUpload\Driver\OpenstackObjectStore;
use Chill\DocStoreBundle\AsyncUpload\Command\ConfigureOpenstackObjectStorageCommand;
use Chill\DocStoreBundle\AsyncUpload\Driver\OpenstackObjectStore\ConfigureOpenstackObjectStorageCommand;
use PHPUnit\Framework\TestCase;
use Symfony\Component\Console\Tester\CommandTester;
use Symfony\Component\DependencyInjection\ParameterBag\ParameterBag;

View File

@@ -9,13 +9,13 @@ declare(strict_types=1);
* the LICENSE file that was distributed with this source code.
*/
namespace Chill\DocStoreBundle\Tests\Service;
namespace Chill\DocStoreBundle\Tests\AsyncUpload\Driver\OpenstackObjectStore;
use Chill\DocStoreBundle\AsyncUpload\Driver\OpenstackObjectStore\StoredObjectManager;
use Chill\DocStoreBundle\AsyncUpload\SignedUrl;
use Chill\DocStoreBundle\AsyncUpload\TempUrlGeneratorInterface;
use Chill\DocStoreBundle\Entity\StoredObject;
use Chill\DocStoreBundle\Exception\StoredObjectManagerException;
use Chill\DocStoreBundle\Service\StoredObjectManager;
use Chill\DocStoreBundle\Service\StoredObjectManagerInterface;
use PHPUnit\Framework\TestCase;
use Symfony\Component\HttpClient\Exception\TransportException;
@@ -27,7 +27,7 @@ use Symfony\Contracts\HttpClient\HttpClientInterface;
/**
* @internal
*
* @covers \Chill\DocStoreBundle\Service\StoredObjectManager
* @covers \Chill\DocStoreBundle\AsyncUpload\Driver\OpenstackObjectStore\StoredObjectManager
*/
final class StoredObjectManagerTest extends TestCase
{

View File

@@ -0,0 +1,338 @@
<?php
declare(strict_types=1);
/*
* Chill is a software for social workers
*
* For the full copyright and license information, please view
* the LICENSE file that was distributed with this source code.
*/
namespace Chill\DocStoreBundle\Tests\Controller;
use Chill\DocStoreBundle\AsyncUpload\Driver\LocalStorage\StoredObjectManager;
use Chill\DocStoreBundle\AsyncUpload\Driver\LocalStorage\TempUrlLocalStorageGenerator;
use Chill\DocStoreBundle\Controller\StoredObjectContentToLocalStorageController;
use PHPUnit\Framework\TestCase;
use Prophecy\Argument;
use Prophecy\PhpUnit\ProphecyTrait;
use Symfony\Component\HttpFoundation\File\UploadedFile;
use Symfony\Component\HttpFoundation\Request;
use Symfony\Component\HttpKernel\Exception\AccessDeniedHttpException;
use Symfony\Component\HttpKernel\Exception\BadRequestHttpException;
use Symfony\Component\HttpKernel\Exception\NotFoundHttpException;
/**
* @internal
*
* @coversNothing
*/
class StoredObjectContentToLocalStorageControllerTest extends TestCase
{
use ProphecyTrait;
/**
* @dataProvider generateOperateContentWithExceptionDataProvider
*/
public function testOperateContentWithException(Request $request, string $expectedException, string $expectedExceptionMessage, bool $existContent, string $readContent, bool $signatureValidity): void
{
$storedObjectManager = $this->prophesize(StoredObjectManager::class);
$storedObjectManager->existsContent(Argument::any())->willReturn($existContent);
$storedObjectManager->readContent(Argument::any())->willReturn($readContent);
$tempUrlLocalStorageGenerator = $this->prophesize(TempUrlLocalStorageGenerator::class);
$tempUrlLocalStorageGenerator->validateSignature(
$request->query->get('sig', ''),
$request->getMethod(),
$request->query->get('object_name', ''),
$request->query->getInt('exp', 0)
)
->willReturn($signatureValidity);
$this->expectException($expectedException);
$this->expectExceptionMessage($expectedExceptionMessage);
$controller = new StoredObjectContentToLocalStorageController(
$storedObjectManager->reveal(),
$tempUrlLocalStorageGenerator->reveal()
);
$controller->contentOperate($request);
}
public function testOperateContentGetHappyScenario(): void
{
$objectName = 'testABC';
$expiration = new \DateTimeImmutable();
$storedObjectManager = $this->prophesize(StoredObjectManager::class);
$storedObjectManager->existsContent($objectName)->willReturn(true);
$storedObjectManager->readContent($objectName)->willReturn('123456789');
$tempUrlLocalStorageGenerator = $this->prophesize(TempUrlLocalStorageGenerator::class);
$tempUrlLocalStorageGenerator->validateSignature('signature', 'GET', $objectName, $expiration->getTimestamp())
->shouldBeCalled()
->willReturn(true);
$controller = new StoredObjectContentToLocalStorageController(
$storedObjectManager->reveal(),
$tempUrlLocalStorageGenerator->reveal()
);
$response = $controller->contentOperate(new Request(['object_name' => $objectName, 'sig' => 'signature', 'exp' => $expiration->getTimestamp()]));
self::assertEquals(200, $response->getStatusCode());
self::assertEquals('123456789', $response->getContent());
}
public function testOperateContentHeadHappyScenario(): void
{
$objectName = 'testABC';
$expiration = new \DateTimeImmutable();
$storedObjectManager = $this->prophesize(StoredObjectManager::class);
$storedObjectManager->existsContent($objectName)->willReturn(true);
$storedObjectManager->readContent($objectName)->willReturn('123456789');
$tempUrlLocalStorageGenerator = $this->prophesize(TempUrlLocalStorageGenerator::class);
$tempUrlLocalStorageGenerator->validateSignature('signature', 'HEAD', $objectName, $expiration->getTimestamp())
->shouldBeCalled()
->willReturn(true);
$controller = new StoredObjectContentToLocalStorageController(
$storedObjectManager->reveal(),
$tempUrlLocalStorageGenerator->reveal()
);
$request = new Request(['object_name' => $objectName, 'sig' => 'signature', 'exp' => $expiration->getTimestamp()]);
$request->setMethod('HEAD');
$response = $controller->contentOperate($request);
self::assertEquals(200, $response->getStatusCode());
self::assertEquals('', $response->getContent());
}
public function testPostContentHappyScenario(): void
{
$expiration = 171899000;
$storedObjectManager = $this->prophesize(StoredObjectManager::class);
$storedObjectManager->writeContent('filePrefix/abcSUFFIX', Argument::containingString('fake_encrypted_content'))
->shouldBeCalled();
$tempUrlGenerator = $this->prophesize(TempUrlLocalStorageGenerator::class);
$tempUrlGenerator->validateSignaturePost('signature', 'filePrefix/abc', $expiration, 15_000_000, 1)
->shouldBeCalled()
->willReturn(true);
$controller = new StoredObjectContentToLocalStorageController($storedObjectManager->reveal(), $tempUrlGenerator->reveal());
$request = new Request(
['prefix' => 'filePrefix/abc'],
['signature' => 'signature', 'expires' => $expiration, 'max_file_size' => 15_000_000, 'max_file_count' => 1],
files: [
'filePrefix/abcSUFFIX' => new UploadedFile(
__DIR__.'/data/StoredObjectContentToLocalStorageControllerTest/dummy_file',
'Document.odt',
test: true
),
]
);
$response = $controller->postContent($request);
self::assertEquals(204, $response->getStatusCode());
}
/**
* @dataProvider generatePostContentWithExceptionDataProvider
*/
public function testPostContentWithException(Request $request, bool $isSignatureValid, string $expectedException, string $expectedExceptionMessage): void
{
$storedObjectManager = $this->prophesize(StoredObjectManager::class);
$storedObjectManager->writeContent(Argument::any(), Argument::any())->shouldNotBeCalled();
$tempUrlGenerator = $this->prophesize(TempUrlLocalStorageGenerator::class);
$tempUrlGenerator->validateSignaturePost('signature', Argument::any(), Argument::any(), Argument::any(), Argument::any())
->willReturn($isSignatureValid);
$controller = new StoredObjectContentToLocalStorageController(
$storedObjectManager->reveal(),
$tempUrlGenerator->reveal()
);
$this->expectException($expectedException);
$this->expectExceptionMessage($expectedExceptionMessage);
$controller->postContent($request);
}
public static function generatePostContentWithExceptionDataProvider(): iterable
{
$query = ['prefix' => 'filePrefix/abc'];
$attributes = ['signature' => 'signature', 'expires' => 15088556855, 'max_file_size' => 15_000_000, 'max_file_count' => 1];
$request = new Request([]);
$request->setMethod('POST');
yield [
$request,
true,
BadRequestHttpException::class,
'Prefix parameter is missing',
];
$attrCloned = [...$attributes];
unset($attrCloned['max_file_size']);
$request = new Request($query, $attrCloned);
$request->setMethod('POST');
yield [
$request,
true,
BadRequestHttpException::class,
'Max file size is not set or equal to zero',
];
$attrCloned = [...$attributes];
unset($attrCloned['max_file_count']);
$request = new Request($query, $attrCloned);
$request->setMethod('POST');
yield [
$request,
true,
BadRequestHttpException::class,
'Max file count is not set or equal to zero',
];
$attrCloned = [...$attributes];
unset($attrCloned['expires']);
$request = new Request($query, $attrCloned);
$request->setMethod('POST');
yield [
$request,
true,
BadRequestHttpException::class,
'Expiration is not set or equal to zero',
];
$attrCloned = [...$attributes];
unset($attrCloned['signature']);
$request = new Request($query, $attrCloned);
$request->setMethod('POST');
yield [
$request,
true,
BadRequestHttpException::class,
'Signature is not set or is a blank string',
];
$request = new Request($query, $attributes);
$request->setMethod('POST');
yield [
$request,
false,
AccessDeniedHttpException::class,
'Invalid signature',
];
$request = new Request($query, $attributes, files: []);
$request->setMethod('POST');
yield [
$request,
true,
BadRequestHttpException::class,
'Zero files given',
];
$request = new Request($query, $attributes, files: [
'filePrefix/abcSUFFIX_1' => new UploadedFile(__DIR__.'/data/StoredObjectContentToLocalStorageControllerTest/dummy_file', 'Some content', test: true),
'filePrefix/abcSUFFIX_2' => new UploadedFile(__DIR__.'/data/StoredObjectContentToLocalStorageControllerTest/dummy_file', 'Some content2', test: true),
]);
$request->setMethod('POST');
yield [
$request,
true,
AccessDeniedHttpException::class,
'More files than max file count',
];
$request = new Request($query, [...$attributes, 'max_file_size' => 3], files: [
'filePrefix/abcSUFFIX_1' => new UploadedFile(__DIR__.'/data/StoredObjectContentToLocalStorageControllerTest/dummy_file', 'Some content', test: true),
]);
$request->setMethod('POST');
yield [
$request,
true,
AccessDeniedHttpException::class,
'File is too big',
];
$request = new Request($query, [...$attributes], files: [
'some/other/prefix_SUFFIX' => new UploadedFile(__DIR__.'/data/StoredObjectContentToLocalStorageControllerTest/dummy_file', 'Some content', test: true),
]);
$request->setMethod('POST');
yield [
$request,
true,
AccessDeniedHttpException::class,
'Filename does not start with signed prefix',
];
}
public static function generateOperateContentWithExceptionDataProvider(): iterable
{
yield [
new Request(['object_name' => '', 'sig' => '', 'exp' => 0]),
BadRequestHttpException::class,
'Object name parameter is missing',
false,
'',
false,
];
yield [
new Request(['object_name' => 'testABC', 'sig' => '', 'exp' => 0]),
BadRequestHttpException::class,
'Expiration is not set or equal to zero',
false,
'',
false,
];
yield [
new Request(['object_name' => 'testABC', 'sig' => '', 'exp' => (new \DateTimeImmutable())->getTimestamp()]),
BadRequestHttpException::class,
'Signature is not set or is a blank string',
false,
'',
false,
];
yield [
new Request(['object_name' => 'testABC', 'sig' => '1234', 'exp' => (new \DateTimeImmutable())->getTimestamp()]),
AccessDeniedHttpException::class,
'Invalid signature',
false,
'',
false,
];
yield [
new Request(['object_name' => 'testABC', 'sig' => '1234', 'exp' => (new \DateTimeImmutable())->getTimestamp()]),
NotFoundHttpException::class,
'Object does not exists on disk',
false,
'',
true,
];
}
}

View File

@@ -0,0 +1,47 @@
<?php
declare(strict_types=1);
/*
* Chill is a software for social workers
*
* For the full copyright and license information, please view
* the LICENSE file that was distributed with this source code.
*/
namespace Chill\DocStoreBundle\Tests\Service\Cryptography;
use Chill\DocStoreBundle\Service\Cryptography\KeyGenerator;
use PHPUnit\Framework\TestCase;
/**
* @internal
*
* @coversNothing
*/
class KeyGeneratorTest extends TestCase
{
public function testGenerateKey(): void
{
$keyGenerator = new KeyGenerator();
$key = $keyGenerator->generateKey();
self::assertNotEmpty($key['k']);
self::assertEquals('A256CBC', $key['alg']);
}
public function testGenerateIv(): void
{
$keyGenerator = new KeyGenerator();
$actual = $keyGenerator->generateIv();
self::assertCount(16, $actual);
foreach ($actual as $value) {
self::assertIsInt($value);
self::assertGreaterThanOrEqual(0, $value);
self::assertLessThan(256, $value);
}
}
}

View File

@@ -8,7 +8,6 @@ services:
tags:
- { name: doctrine.repository_service }
Chill\DocStoreBundle\Security\Authorization\:
resource: "./../Security/Authorization"
@@ -51,11 +50,9 @@ services:
Chill\DocStoreBundle\AsyncUpload\Driver\:
resource: '../AsyncUpload/Driver/'
Chill\DocStoreBundle\AsyncUpload\Driver\LocalStorage\TempUrlLocalStorageGenerator:
arguments:
$secret: '%kernel.secret%'
Chill\DocStoreBundle\AsyncUpload\Templating\:
resource: '../AsyncUpload/Templating/'
Chill\DocStoreBundle\AsyncUpload\Command\:
resource: '../AsyncUpload/Command/'
Chill\DocStoreBundle\AsyncUpload\TempUrlGeneratorInterface:
alias: Chill\DocStoreBundle\AsyncUpload\Driver\OpenstackObjectStore\TempUrlOpenstackGenerator

View File

@@ -16,6 +16,7 @@ use Chill\MainBundle\Service\Import\PostalCodeBEFromBestAddress;
use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Input\InputArgument;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Input\InputOption;
use Symfony\Component\Console\Output\OutputInterface;
class LoadAddressesBEFromBestAddressCommand extends Command
@@ -34,14 +35,19 @@ class LoadAddressesBEFromBestAddressCommand extends Command
$this
->setName('chill:main:address-ref-from-best-addresses')
->addArgument('lang', InputArgument::REQUIRED, "Language code, for example 'fr'")
->addArgument('list', InputArgument::IS_ARRAY, "The list to add, for example 'full', or 'extract' (dev) or '1xxx' (brussel CP)");
->addArgument('list', InputArgument::IS_ARRAY, "The list to add, for example 'full', or 'extract' (dev) or '1xxx' (brussel CP)")
->addOption('send-report-email', 's', InputOption::VALUE_REQUIRED, 'Email address where a list of unimported addresses can be send');
}
protected function execute(InputInterface $input, OutputInterface $output): int
{
$this->postalCodeBEFromBestAddressImporter->import();
$this->addressImporter->import($input->getArgument('lang'), $input->getArgument('list'));
$this->addressImporter->import(
$input->getArgument('lang'),
$input->getArgument('list'),
$input->hasOption('send-report-email') ? $input->getOption('send-report-email') : null
);
return Command::SUCCESS;
}

View File

@@ -0,0 +1,48 @@
<?php
declare(strict_types=1);
/*
* Chill is a software for social workers
*
* For the full copyright and license information, please view
* the LICENSE file that was distributed with this source code.
*/
namespace Chill\MainBundle\Command;
use Chill\MainBundle\Service\Import\AddressReferenceFromBAN;
use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Input\InputArgument;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Input\InputOption;
use Symfony\Component\Console\Output\OutputInterface;
class LoadAddressesFRFromBANCommand extends Command
{
protected static $defaultDescription = 'Import FR addresses from BAN (see https://adresses.data.gouv.fr';
public function __construct(private readonly AddressReferenceFromBAN $addressReferenceFromBAN)
{
parent::__construct();
}
protected function configure()
{
$this->setName('chill:main:address-ref-from-ban')
->addArgument('departementNo', InputArgument::REQUIRED | InputArgument::IS_ARRAY, 'a list of departement numbers')
->addOption('send-report-email', 's', InputOption::VALUE_REQUIRED, 'Email address where a list of unimported addresses can be send');
}
protected function execute(InputInterface $input, OutputInterface $output): int
{
dump(__METHOD__);
foreach ($input->getArgument('departementNo') as $departementNo) {
$output->writeln('Import addresses for '.$departementNo);
$this->addressReferenceFromBAN->import($departementNo, $input->hasOption('send-report-email') ? $input->getOption('send-report-email') : null);
}
return Command::SUCCESS;
}
}

View File

@@ -15,6 +15,7 @@ use Chill\MainBundle\Service\Import\AddressReferenceFromBano;
use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Input\InputArgument;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Input\InputOption;
use Symfony\Component\Console\Output\OutputInterface;
class LoadAddressesFRFromBANOCommand extends Command
@@ -29,7 +30,8 @@ class LoadAddressesFRFromBANOCommand extends Command
protected function configure()
{
$this->setName('chill:main:address-ref-from-bano')
->addArgument('departementNo', InputArgument::REQUIRED | InputArgument::IS_ARRAY, 'a list of departement numbers');
->addArgument('departementNo', InputArgument::REQUIRED | InputArgument::IS_ARRAY, 'a list of departement numbers')
->addOption('send-report-email', 's', InputOption::VALUE_REQUIRED, 'Email address where a list of unimported addresses can be send');
}
protected function execute(InputInterface $input, OutputInterface $output): int
@@ -37,7 +39,7 @@ class LoadAddressesFRFromBANOCommand extends Command
foreach ($input->getArgument('departementNo') as $departementNo) {
$output->writeln('Import addresses for '.$departementNo);
$this->addressReferenceFromBano->import($departementNo);
$this->addressReferenceFromBano->import($departementNo, $input->hasOption('send-report-email') ? $input->getOption('send-report-email') : null);
}
return Command::SUCCESS;

View File

@@ -14,6 +14,7 @@ namespace Chill\MainBundle\Command;
use Chill\MainBundle\Service\Import\AddressReferenceLU;
use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Input\InputOption;
use Symfony\Component\Console\Output\OutputInterface;
class LoadAddressesLUFromBDAddressCommand extends Command
@@ -28,12 +29,16 @@ class LoadAddressesLUFromBDAddressCommand extends Command
protected function configure()
{
$this->setName('chill:main:address-ref-lux');
$this
->setName('chill:main:address-ref-lux')
->addOption('send-report-email', 's', InputOption::VALUE_REQUIRED, 'Email address where a list of unimported addresses can be send');
}
protected function execute(InputInterface $input, OutputInterface $output): int
{
$this->addressImporter->import();
$this->addressImporter->import(
$input->hasOption('send-report-email') ? $input->getOption('send-report-email') : null,
);
return Command::SUCCESS;
}

View File

@@ -95,7 +95,6 @@ class UserController extends CRUDController
return $this->render('@ChillMain/User/edit.html.twig', [
'entity' => $user,
'access_permissions_group_list' => $this->parameterBag->get('chill_main.access_permissions_group_list'),
'edit_form' => $this->createEditForm($user)->createView(),
'add_groupcenter_form' => $this->createAddLinkGroupCenterForm($user, $request)->createView(),
'delete_groupcenter_form' => array_map(
static fn (Form $form) => $form->createView(),

View File

@@ -230,7 +230,7 @@ abstract class AbstractWidgetsCompilerPass implements CompilerPassInterface
// check the alias does not exists yet
if (\array_key_exists($attr[self::WIDGET_SERVICE_TAG_ALIAS], $this->widgetServices)) {
throw new InvalidArgumentException('a service has already be defined with the '.self::WIDGET_SERVICE_TAG_ALIAS.' '.$attr[self::WIDGET_SERVICE_TAG_ALIAS]);
throw new InvalidConfigurationException('a service has already be defined with the '.self::WIDGET_SERVICE_TAG_ALIAS.' '.$attr[self::WIDGET_SERVICE_TAG_ALIAS]);
}
// register the service as available
@@ -259,7 +259,7 @@ abstract class AbstractWidgetsCompilerPass implements CompilerPassInterface
// check the alias does not exists yet
if (\array_key_exists($alias, $this->widgetServices)) {
throw new InvalidArgumentException('a service has already be defined with the '.self::WIDGET_SERVICE_TAG_ALIAS.' '.$alias);
throw new InvalidConfigurationException('a service has already be defined with the '.self::WIDGET_SERVICE_TAG_ALIAS.' '.$alias);
}
// register the factory as available

View File

@@ -28,8 +28,8 @@ class EntityToJsonTransformer implements DataTransformerInterface
public function reverseTransform($value)
{
if (false === $this->multiple && '' === $value) {
return null;
if ('' === $value) {
return $this->multiple ? [] : null;
}
if ($this->multiple && [] === $value) {

View File

@@ -22,10 +22,10 @@ class AddressReferenceBEFromBestAddress
public function __construct(private readonly HttpClientInterface $client, private readonly AddressReferenceBaseImporter $baseImporter, private readonly AddressToReferenceMatcher $addressToReferenceMatcher) {}
public function import(string $lang, array $lists): void
public function import(string $lang, array $lists, ?string $sendAddressReportToEmail = null): void
{
foreach ($lists as $list) {
$this->importList($lang, $list);
$this->importList($lang, $list, $sendAddressReportToEmail);
}
}
@@ -43,7 +43,7 @@ class AddressReferenceBEFromBestAddress
return array_values($asset)[0]['browser_download_url'];
}
private function importList(string $lang, string $list): void
private function importList(string $lang, string $list, ?string $sendAddressReportToEmail = null): void
{
$downloadUrl = $this->getDownloadUrl($lang, $list);
@@ -85,7 +85,7 @@ class AddressReferenceBEFromBestAddress
);
}
$this->baseImporter->finalize();
$this->baseImporter->finalize(sendAddressReportToEmail: $sendAddressReportToEmail);
$this->addressToReferenceMatcher->checkAddressesMatchingReferences();

View File

@@ -13,7 +13,12 @@ namespace Chill\MainBundle\Service\Import;
use Doctrine\DBAL\Connection;
use Doctrine\DBAL\Statement;
use League\Csv\Writer;
use Psr\Log\LoggerInterface;
use Symfony\Component\Filesystem\Path;
use Symfony\Component\Mailer\Exception\TransportExceptionInterface;
use Symfony\Component\Mailer\MailerInterface;
use Symfony\Component\Mime\Email;
/**
* Import addresses into the database.
@@ -25,15 +30,15 @@ final class AddressReferenceBaseImporter
{
private const INSERT = <<<'SQL'
INSERT INTO reference_address_temp
(postcode_id, refid, street, streetnumber, municipalitycode, source, point)
(postcode_id, postalcode, refid, street, streetnumber, municipalitycode, source, point)
SELECT
cmpc.id, i.refid, i.street, i.streetnumber, i.refpostalcode, i.source,
cmpc.id, i.postalcode, i.refid, i.street, i.streetnumber, i.refpostalcode, i.source,
CASE WHEN (i.lon::float != 0.0 AND i.lat::float != 0.0) THEN ST_Transform(ST_setSrid(ST_point(i.lon::float, i.lat::float), i.srid::int), 4326) ELSE NULL END
FROM
(VALUES
{{ values }}
) AS i (refid, refpostalcode, postalcode, street, streetnumber, source, lat, lon, srid)
JOIN chill_main_postal_code cmpc ON cmpc.refpostalcodeid = i.refpostalcode and cmpc.code = i.postalcode
LEFT JOIN chill_main_postal_code cmpc ON cmpc.refpostalcodeid = i.refpostalcode and cmpc.code = i.postalcode
SQL;
private const LOG_PREFIX = '[AddressReferenceImporter] ';
@@ -51,7 +56,11 @@ final class AddressReferenceBaseImporter
private array $waitingForInsert = [];
public function __construct(private readonly Connection $defaultConnection, private readonly LoggerInterface $logger) {}
public function __construct(
private readonly Connection $defaultConnection,
private readonly LoggerInterface $logger,
private readonly MailerInterface $mailer,
) {}
/**
* Finalize the import process and make reconciliation with addresses.
@@ -60,11 +69,11 @@ final class AddressReferenceBaseImporter
*
* @throws \Exception
*/
public function finalize(bool $allowRemoveDoubleRefId = false): void
public function finalize(bool $allowRemoveDoubleRefId = false, ?string $sendAddressReportToEmail = null): void
{
$this->doInsertPending();
$this->updateAddressReferenceTable($allowRemoveDoubleRefId);
$this->updateAddressReferenceTable($allowRemoveDoubleRefId, $sendAddressReportToEmail);
$this->deleteTemporaryTable();
@@ -116,7 +125,8 @@ final class AddressReferenceBaseImporter
private function createTemporaryTable(): void
{
$this->defaultConnection->executeStatement('CREATE TEMPORARY TABLE reference_address_temp (
postcode_id INT,
postcode_id INT DEFAULT NULL,
postalcode TEXT DEFAULT \'\',
refid VARCHAR(255),
street VARCHAR(255),
streetnumber VARCHAR(255),
@@ -185,15 +195,15 @@ final class AddressReferenceBaseImporter
$this->isInitialized = true;
}
private function updateAddressReferenceTable(bool $allowRemoveDoubleRefId): void
private function updateAddressReferenceTable(bool $allowRemoveDoubleRefId, ?string $sendAddressReportToEmail = null): void
{
$this->defaultConnection->executeStatement(
'CREATE INDEX idx_ref_add_temp ON reference_address_temp (refid)'
'CREATE INDEX idx_ref_add_temp ON reference_address_temp (refid) WHERE postcode_id IS NOT NULL'
);
// 0) detect for doublon in current temporary table
$results = $this->defaultConnection->executeQuery(
'SELECT COUNT(*) AS nb_appearance, refid FROM reference_address_temp GROUP BY refid HAVING count(*) > 1'
'SELECT COUNT(*) AS nb_appearance, refid FROM reference_address_temp WHERE postcode_id IS NOT NULL GROUP BY refid HAVING count(*) > 1'
);
$hasDouble = false;
@@ -210,7 +220,7 @@ final class AddressReferenceBaseImporter
WITH ordering AS (
SELECT gid, rank() over (PARTITION BY refid ORDER BY gid DESC) AS ranking
FROM reference_address_temp
WHERE refid IN (SELECT refid FROM reference_address_temp group by refid having count(*) > 1)
WHERE postcode_id IS NOT NULL AND refid IN (SELECT refid FROM reference_address_temp WHERE postcode_id IS NOT NULL group by refid having count(*) > 1)
),
keep_last AS (
SELECT gid, ranking FROM ordering where ranking > 1
@@ -240,7 +250,7 @@ final class AddressReferenceBaseImporter
NOW(),
null,
NOW()
FROM reference_address_temp
FROM reference_address_temp WHERE postcode_id IS NOT NULL
ON CONFLICT (refid, source) DO UPDATE
SET postcode_id = excluded.postcode_id, refid = excluded.refid, street = excluded.street, streetnumber = excluded.streetnumber, municipalitycode = excluded.municipalitycode, source = excluded.source, point = excluded.point, updatedat = NOW(), deletedAt = NULL
");
@@ -251,10 +261,65 @@ final class AddressReferenceBaseImporter
$affected = $connection->executeStatement('UPDATE chill_main_address_reference
SET deletedat = NOW()
WHERE
chill_main_address_reference.refid NOT IN (SELECT refid FROM reference_address_temp WHERE source LIKE ?)
chill_main_address_reference.refid NOT IN (SELECT refid FROM reference_address_temp WHERE source LIKE ? AND postcode_id IS NOT NULL)
AND chill_main_address_reference.source LIKE ?
', [$this->currentSource, $this->currentSource]);
$this->logger->info(self::LOG_PREFIX.'addresses deleted', ['deleted' => $affected]);
});
// Create a list of addresses without any postal code
$results = $this->defaultConnection->executeQuery('SELECT
postalcode,
refid,
street,
streetnumber,
municipalitycode,
source,
ST_AsText(point)
FROM reference_address_temp
WHERE postcode_id IS NULL
');
$count = $results->rowCount();
if ($count > 0) {
$this->logger->warning(self::LOG_PREFIX.'There are addresses that could not be associated with a postal code', ['nb' => $count]);
$filename = sprintf('%s-%s.csv', (new \DateTimeImmutable())->format('Ymd-His'), uniqid());
$path = Path::normalize(sprintf('%s%s%s', sys_get_temp_dir(), DIRECTORY_SEPARATOR, $filename));
$writer = Writer::createFromPath($path, 'w+');
// insert headers
$writer->insertOne([
'postalcode',
'refid',
'street',
'streetnumber',
'municipalitycode',
'source',
'point',
]);
$writer->insertAll($results->iterateAssociative());
$this->logger->info(sprintf(self::LOG_PREFIX.'The addresses that could not be inserted within the database are registered at path %s', $path));
if (null !== $sendAddressReportToEmail) {
// first, we compress the existing file which can be quite big
$attachment = gzopen($attachmentPath = sprintf('%s.gz', $path), 'w9');
gzwrite($attachment, file_get_contents($path));
gzclose($attachment);
$email = (new Email())
->addTo($sendAddressReportToEmail)
->subject('Addresses that could not be imported')
->attach(file_get_contents($attachmentPath), sprintf('%s.gz', $path));
try {
$this->mailer->send($email);
} catch (TransportExceptionInterface $e) {
$this->logger->error(self::LOG_PREFIX.'Could not send an email with addresses that could not be registered', ['exception' => $e->getTraceAsString()]);
}
unlink($attachmentPath);
}
}
}
}

View File

@@ -0,0 +1,106 @@
<?php
declare(strict_types=1);
/*
* Chill is a software for social workers
*
* For the full copyright and license information, please view
* the LICENSE file that was distributed with this source code.
*/
namespace Chill\MainBundle\Service\Import;
use League\Csv\Reader;
use League\Csv\Statement;
use Symfony\Contracts\HttpClient\HttpClientInterface;
class AddressReferenceFromBAN
{
public function __construct(
private readonly HttpClientInterface $client,
private readonly AddressReferenceBaseImporter $baseImporter,
private readonly AddressToReferenceMatcher $addressToReferenceMatcher,
) {}
public function import(string $departementNo, ?string $sendAddressReportToEmail = null): void
{
if (!is_numeric($departementNo)) {
throw new \UnexpectedValueException('Could not parse this department number');
}
$url = sprintf('https://adresse.data.gouv.fr/data/ban/adresses/latest/csv/adresses-%s.csv.gz', $departementNo);
$response = $this->client->request('GET', $url);
if (200 !== $response->getStatusCode()) {
throw new \Exception('Could not download CSV: '.$response->getStatusCode());
}
$path = sys_get_temp_dir().'/'.$departementNo.'.csv.gz';
$file = fopen($path, 'w');
if (false === $file) {
throw new \Exception('Could not create temporary file');
}
foreach ($this->client->stream($response) as $chunk) {
fwrite($file, $chunk->getContent());
}
fclose($file);
// re-open it to read it
$csvDecompressed = gzopen($path, 'r');
$csv = Reader::createFromStream($csvDecompressed);
$csv->setDelimiter(';')->setHeaderOffset(0);
$stmt = Statement::create()
->process($csv, [
'id',
'id_fantoir',
'numero',
'rep',
'nom_voie',
'code_postal',
'code_insee',
'nom_commune',
'code_insee_ancienne_commune',
'nom_ancienne_commune',
'x',
'y',
'lon',
'lat',
'type_position',
'alias',
'nom_ld',
'libelle_acheminement',
'nom_afnor',
'source_position',
'source_nom_voie',
'certification_commune',
'cad_parcelles',
]);
foreach ($stmt as $record) {
$this->baseImporter->importAddress(
$record['id'],
$record['code_insee'],
$record['code_postal'],
$record['nom_voie'],
$record['numero'].' '.$record['rep'],
'BAN.'.$departementNo,
(float) $record['lat'],
(float) $record['lon'],
4326
);
}
$this->baseImporter->finalize(sendAddressReportToEmail: $sendAddressReportToEmail);
$this->addressToReferenceMatcher->checkAddressesMatchingReferences();
fclose($csvDecompressed);
unlink($path);
}
}

View File

@@ -19,7 +19,7 @@ class AddressReferenceFromBano
{
public function __construct(private readonly HttpClientInterface $client, private readonly AddressReferenceBaseImporter $baseImporter, private readonly AddressToReferenceMatcher $addressToReferenceMatcher) {}
public function import(string $departementNo): void
public function import(string $departementNo, ?string $sendAddressReportToEmail = null): void
{
if (!is_numeric($departementNo) || !\is_int((int) $departementNo)) {
throw new \UnexpectedValueException('Could not parse this department number');
@@ -69,7 +69,7 @@ class AddressReferenceFromBano
);
}
$this->baseImporter->finalize();
$this->baseImporter->finalize(sendAddressReportToEmail: $sendAddressReportToEmail);
$this->addressToReferenceMatcher->checkAddressesMatchingReferences();

View File

@@ -21,7 +21,7 @@ class AddressReferenceLU
public function __construct(private readonly HttpClientInterface $client, private readonly AddressReferenceBaseImporter $addressBaseImporter, private readonly PostalCodeBaseImporter $postalCodeBaseImporter, private readonly AddressToReferenceMatcher $addressToReferenceMatcher) {}
public function import(): void
public function import(?string $sendAddressReportToEmail = null): void
{
$downloadUrl = self::RELEASE;
@@ -45,14 +45,14 @@ class AddressReferenceLU
$this->process_postal_code($csv);
$this->process_address($csv);
$this->process_address($csv, $sendAddressReportToEmail);
$this->addressToReferenceMatcher->checkAddressesMatchingReferences();
fclose($file);
}
private function process_address(Reader $csv): void
private function process_address(Reader $csv, ?string $sendAddressReportToEmail = null): void
{
$stmt = Statement::create()->process($csv);
foreach ($stmt as $record) {
@@ -69,7 +69,7 @@ class AddressReferenceLU
);
}
$this->addressBaseImporter->finalize();
$this->addressBaseImporter->finalize(sendAddressReportToEmail: $sendAddressReportToEmail);
}
private function process_postal_code(Reader $csv): void

View File

@@ -47,6 +47,12 @@ services:
tags:
- { name: console.command }
Chill\MainBundle\Command\LoadAddressesFRFromBANCommand:
autoconfigure: true
autowire: true
tags:
- { name: console.command }
Chill\MainBundle\Command\LoadAddressesBEFromBestAddressCommand:
autoconfigure: true
autowire: true

View File

@@ -0,0 +1,149 @@
<?php
declare(strict_types=1);
/*
* Chill is a software for social workers
*
* For the full copyright and license information, please view
* the LICENSE file that was distributed with this source code.
*/
namespace Chill\PersonBundle\Controller;
use Chill\PersonBundle\Repository\SocialWork\SocialActionRepository;
use Chill\PersonBundle\Repository\SocialWork\SocialIssueRepository;
use Chill\PersonBundle\Templating\Entity\SocialActionRender;
use Chill\PersonBundle\Templating\Entity\SocialIssueRender;
use League\Csv\CannotInsertRecord;
use League\Csv\Exception;
use League\Csv\UnavailableStream;
use League\Csv\Writer;
use Symfony\Bundle\FrameworkBundle\Controller\AbstractController;
use Symfony\Component\HttpFoundation\Request;
use Symfony\Component\HttpFoundation\Response;
use Symfony\Component\HttpFoundation\StreamedResponse;
use Symfony\Component\HttpKernel\Exception\AccessDeniedHttpException;
use Symfony\Component\Routing\Annotation\Route;
use Symfony\Component\Security\Core\Security;
use Symfony\Contracts\Translation\TranslatorInterface;
class SocialWorkExportController extends AbstractController
{
public function __construct(
private readonly SocialIssueRepository $socialIssueRepository,
private readonly SocialActionRepository $socialActionRepository,
private readonly Security $security,
private readonly TranslatorInterface $translator,
private readonly SocialIssueRender $socialIssueRender,
private readonly SocialActionRender $socialActionRender,
) {}
/**
* @throws UnavailableStream
* @throws CannotInsertRecord
* @throws Exception
*/
#[Route(path: '/{_locale}/admin/social-work/social-issue/export/list.{_format}', name: 'chill_person_social_issue_export_list', requirements: ['_format' => 'csv'])]
public function socialIssueList(Request $request, string $_format = 'csv'): StreamedResponse
{
if (!$this->security->isGranted('ROLE_ADMIN')) {
throw new AccessDeniedHttpException('Only ROLE_ADMIN can export this list');
}
$socialIssues = $this->socialIssueRepository->findAll();
$socialIssues = array_map(fn ($issue) => [
'id' => $issue->getId(),
'title' => $this->socialIssueRender->renderString($issue, []),
'ordering' => $issue->getOrdering(),
'desactivationDate' => $issue->getDesactivationDate(),
], $socialIssues);
$csv = Writer::createFromPath('php://temp', 'r+');
$csv->insertOne(
array_map(
fn (string $e) => $this->translator->trans($e),
[
'Id',
'Title',
'Ordering',
'goal.desactivationDate',
]
)
);
$csv->addFormatter(fn (array $row) => null !== ($row['desactivationDate'] ?? null) ? array_merge($row, ['desactivationDate' => $row['desactivationDate']->format('Y-m-d')]) : $row);
$csv->insertAll($socialIssues);
return new StreamedResponse(
function () use ($csv) {
foreach ($csv->chunk(1024) as $chunk) {
echo $chunk;
flush();
}
},
Response::HTTP_OK,
[
'Content-Encoding' => 'none',
'Content-Type' => 'text/csv; charset=UTF-8',
'Content-Disposition' => 'attachment; users.csv',
]
);
}
/**
* @throws UnavailableStream
* @throws CannotInsertRecord
* @throws Exception
*/
#[Route(path: '/{_locale}/admin/social-work/social-action/export/list.{_format}', name: 'chill_person_social_action_export_list', requirements: ['_format' => 'csv'])]
public function socialActionList(Request $request, string $_format = 'csv'): StreamedResponse
{
if (!$this->security->isGranted('ROLE_ADMIN')) {
throw new AccessDeniedHttpException('Only ROLE_ADMIN can export this list');
}
$socialActions = $this->socialActionRepository->findAll();
$socialActions = array_map(fn ($action) => [
'id' => $action->getId(),
'title' => $this->socialActionRender->renderString($action, []),
'desactivationDate' => $action->getDesactivationDate(),
'socialIssue' => $this->socialIssueRender->renderString($action->getIssue(), []),
'ordering' => $action->getOrdering(),
], $socialActions);
$csv = Writer::createFromPath('php://temp', 'r+');
$csv->insertOne(
array_map(
fn (string $e) => $this->translator->trans($e),
[
'Id',
'Title',
'goal.desactivationDate',
'Social issue',
'Ordering',
]
)
);
$csv->addFormatter(fn (array $row) => null !== ($row['desactivationDate'] ?? null) ? array_merge($row, ['desactivationDate' => $row['desactivationDate']->format('Y-m-d')]) : $row);
$csv->insertAll($socialActions);
return new StreamedResponse(
function () use ($csv) {
foreach ($csv->chunk(1024) as $chunk) {
echo $chunk;
flush();
}
},
Response::HTTP_OK,
[
'Content-Encoding' => 'none',
'Content-Type' => 'text/csv; charset=UTF-8',
'Content-Disposition' => 'attachment; users.csv',
]
);
}
}

View File

@@ -35,6 +35,9 @@
{% endblock %}
{% block actions_before %}
<li>
<a href="{{ path('chill_person_social_action_export_list') }}" class="btn btn-download"></a>
</li>
<li class='cancel'>
<a href="{{ path('chill_main_admin_central') }}" class="btn btn-cancel">{{'Back to the admin'|trans}}</a>
</li>

View File

@@ -35,6 +35,9 @@
{% endblock %}
{% block actions_before %}
<li>
<a href="{{ path('chill_person_social_issue_export_list') }}" class="btn btn-download"></a>
</li>
<li class='cancel'>
<a href="{{ path('chill_main_admin_central') }}" class="btn btn-cancel">{{'Back to the admin'|trans}}</a>
</li>

View File

@@ -71,3 +71,6 @@ services:
Chill\PersonBundle\Controller\PersonSignatureController:
tags: [ 'controller.service_arguments' ]
Chill\PersonBundle\Controller\SocialWorkExportController:
tags: [ 'controller.service_arguments' ]

View File

@@ -13,7 +13,6 @@ namespace Chill\WopiBundle\Controller;
use Chill\DocStoreBundle\Entity\StoredObject;
use Chill\DocStoreBundle\Security\Authorization\StoredObjectRoleEnum;
use Chill\DocStoreBundle\Service\StoredObjectManager;
use Chill\DocStoreBundle\Service\StoredObjectManagerInterface;
use Chill\WopiBundle\Service\WopiConverter;
use Psr\Log\LoggerInterface;
@@ -26,9 +25,6 @@ class ConvertController
{
private const LOG_PREFIX = '[convert] ';
/**
* @param StoredObjectManager $storedObjectManager
*/
public function __construct(
private readonly Security $security,
private readonly StoredObjectManagerInterface $storedObjectManager,

1053
yarn.lock

File diff suppressed because it is too large Load Diff