Unlimited .data fields

cyrill.lippuner · 21 Sept 2022, 16:11

Hello,

This is not a direct bug, but maybe a safety net for other users to not do the same mistake as we did.

We had a prod environment going down due to fusionauth OOM errors which were cause by a bug on one of our services. There is everything fine with the FusionAuth in general, but the problem was that we filled up the users.data field for each user with too much data due to an error (should only have been a list of some bytes). Therefore, after some months we started to have some occasional OOM errors of FusionAuth, as the 0.5GB RAM were not sufficient anymore to load even a single user (which had a users.data text field of 400MB).

After cleaning that, everything is back to normal.

My proposition might be, to put a (maybe configurable) size limit on the *.data fields to prevent such hard to catch runtime errors.

Feel free to ask back for more info, I just wanted to put this here in case you might wanna consider it

dan · 22 Sept 2022, 12:45

@cyrill-lippuner Thanks for sharing this info with the community!

Did you also explore increasing the memory available to FusionAuth? That would be another approach, if someone actually really needed a large amount of data in user.data.

However, 400MB is a pretty big JSON blob!

If you'd like to file a feature request about limiting user.data size, that'd be awesome.

cyrill.lippuner · 27 Sept 2022, 16:32

@dan Yes I did increase it until 2GB, but then loading a list of 4 users also fails ^^

So I think it is just not a good idea using FA as a database

Will look into the feature request.

dan · 27 Sept 2022, 21:15

@cyrill-lippuner Thanks for filing the feature request: https://github.com/FusionAuth/fusionauth-issues/issues/1901

john.laflamme · 4 Oct 2023, 13:30

@cyrill-lippuner said in Unlimited .data fields:

Hello,
This is not a direct bug, but maybe a safety net for other users to not do the same mistake as we did.
We had a prod environment going down due to fusionauth OOM errors which were cause by a bug on one of our services. There is everything fine with the FusionAuth in general, but the problem was that we filled up the users.data field for each user with too much data due to an error (should only have been a list of some bytes). Therefore, after some months we started to have some occasional OOM errors of FusionAuth, as the 0.5GB RAM were not sufficient anymore to load even a single user (which had a users.data text field of 400MB).
After cleaning that, everything is back to normal.
My proposition might be, to put a (maybe configurable) size limit on the *.data fields to prevent such hard to catch runtime errors.
Feel free to ask back for more info, I just wanted to put this here in case you might wanna consider it

Implementing a configurable size limit on data fields is indeed a useful safety measure. This limit can help prevent unintended data growth and ensure that the system behaves predictably even when users inadvertently input excessive data.