Discussions
Categories
- 197K All Categories
- 2.5K Data
- 546 Big Data Appliance
- 1.9K Data Science
- 450.8K Databases
- 221.9K General Database Discussions
- 3.8K Java and JavaScript in the Database
- 31 Multilingual Engine
- 552 MySQL Community Space
- 479 NoSQL Database
- 7.9K Oracle Database Express Edition (XE)
- 3.1K ORDS, SODA & JSON in the Database
- 556 SQLcl
- 4K SQL Developer Data Modeler
- 187.2K SQL & PL/SQL
- 21.4K SQL Developer
- 296.4K Development
- 17 Developer Projects
- 139 Programming Languages
- 293.1K Development Tools
- 111 DevOps
- 3.1K QA/Testing
- 646.1K Java
- 28 Java Learning Subscription
- 37K Database Connectivity
- 161 Java Community Process
- 105 Java 25
- 22.1K Java APIs
- 138.2K Java Development Tools
- 165.3K Java EE (Java Enterprise Edition)
- 19 Java Essentials
- 162 Java 8 Questions
- 86K Java Programming
- 81 Java Puzzle Ball
- 65.1K New To Java
- 1.7K Training / Learning / Certification
- 13.8K Java HotSpot Virtual Machine
- 94.3K Java SE
- 13.8K Java Security
- 205 Java User Groups
- 24 JavaScript - Nashorn
- Programs
- 475 LiveLabs
- 39 Workshops
- 10.2K Software
- 6.7K Berkeley DB Family
- 3.5K JHeadstart
- 5.7K Other Languages
- 2.3K Chinese
- 175 Deutsche Oracle Community
- 1.1K Español
- 1.9K Japanese
- 233 Portuguese
Oracle.DataAccess.Client.OracleException (0x80004005): Memory could not be allocated
A client application is generating the error:
Oracle.DataAccess.Client.OracleException (0x80004005): Memory could not be allocated
and the db update is failing. I've checked the memory for the process (fine; steady at about 3% of the machine's memory) and the vm as a whole (steady at about 50% usage.) It happens intermittently - individual transactions, or occasionally a couple in a row will fail, and then the next will succeed again.
Database server is running 19c. DataAccess client library dll used by the application is:
C:\WINDOWS\Microsoft.Net\assembly\GAC_32\Oracle.DataAccess\v4.0_4.122.19.1__89b483f429c47342\Oracle.DataAccess.dll
I'm wondering if anyone can shed some light on why this error is being generated, and what I can do about it?
Thanks for any assistance,
- rob.
Best Answers
-
I noticed your ODP.NET is in the 32-bit GAC. If your app is 32-bit, then a process can maximally access 2 GB, 3 GB if you explicitly enable that. While your 64-bit OS and VM may indicate there's a ton of memory available, that memory may not be available to your 32-bit app.
I looked through the Oracle DB bug DB, but did not find any existing ODP.NET bugs that fit your symptoms. If's not a 32-bit issue, it's possible another part of the Oracle Client layer, outside of ODP.NET has a bug. There may not be a bug, but a confluence of factors in your specific client machine setup that causes the issue.
-
I've proved to myself that it is indeed the 2GB limit for 32-bit processes that is causing the memory error, so I'm going to call this solved, and leave it here in case someone else finds it useful.
Haven't yet worked out where the memory leak in our code is, but that's not really Oracle's problem...
Answers
-
Are the updates with the error particularly large? I've seen this error happen when ODP.NET can't allocate a contiguous amount of memory from the machine, generally it's a large amount. It's possible that with 50% of the VM memory used that there is enough fragmentation that makes it harder to find large contiguous amounts of memory.
-
Not particularly, no. Good question, and I wondered about this myself, so I did a bit of statistical analysis on the updates that fail with that error vs the ones that succeed. The table we're adding a new row to has a number of small columns and one big, open-ended "comments" column, which is often null but occasionally larger than the rest of the data combined. There is no correlation between the size (or presence) of that comments column (or any other column, though that one seemed the most significant) and whether the insert fails or succeeds.
Given the size of the smallest updates that fail, and the amount of total memory free on the system, it seems unlikely but not impossible that fragmentation is the issue. Any suggestions on how to detect that it is or isn't, and or resolve the fragmentation? There's one primary application running on the servers; I could just try a regular restart of that application?
-
I was able to get an additional layer of application logging going, which shows me the exact query being run at the time the error was generated... and its not what I thought.
The application attempts to insert a new row in the table, but thats not where its failing. Before the insert it runs a user-defined database function by running a query like:
SELECT function1(a, b, c) as result from dual
The memory allocation failures are always on these function call queries (but not all of the function call queries generate the error; only about 10%. And as previously mentioned it doesn't appear to have anything to do with time, or the inputs to the function.) So it seems natural to suspect something awry with this function. Is there any way to track the memory usage of the function calls themselves? Or the session that the function call queries are running in?
-
I noticed your ODP.NET is in the 32-bit GAC. If your app is 32-bit, then a process can maximally access 2 GB, 3 GB if you explicitly enable that. While your 64-bit OS and VM may indicate there's a ton of memory available, that memory may not be available to your 32-bit app.
I looked through the Oracle DB bug DB, but did not find any existing ODP.NET bugs that fit your symptoms. If's not a 32-bit issue, it's possible another part of the Oracle Client layer, outside of ODP.NET has a bug. There may not be a bug, but a confluence of factors in your specific client machine setup that causes the issue.
-
An excellent point that I hadn't considered. I'll have a look. Thanks!
-
Coming back to this after a bit. It is a 32-bit app, but for various reasons must remain so. It still doesn't really make sense that the function in question would need even that much memory. But I did find that the algorithm it was running was a bit inefficient, so I wrote up a better version in hopes that that would help, and I've been waiting all this time for it to make it's way through to production, as we've never managed to reproduce this problem in test.
It made it to prod. It didn't help. Any other ideas?
Still seems to 100% consistently happen while running a particular database function. If there is any way to monitor the memory used during calls to a specific function, that would be handy.
-
I should mention that I'd be happy to post a copy of the function here if anyone thinks that would be helpful. The only reason I haven't, is that it's just not behaving like the memory usage of an individual function call was the issue. The function takes a map coordinate, and finds the nearest point of interest in the database to that coordinate. So multiple calls to the function with the same inputs should go through exactly the same process, and return the same result. If a single function call was going mad and causing us to hit the limit for a 32-bit process, I would expect any repeat call with the same inputs to also hit the limit. This is not the case. The failures also don't necessarily seem to clump by time, which I might expect if it was more related to the process memory usage. Nor are the calls coming particularly quickly. I can see from the logs:
A successful call
A minute or so later, the next call fails with the memory error.
A minute or so later, the next call is successful.
Some time later, a successful call, on exactly the same input coordinates as the earlier failed call.
-
It's been a long time since I brushed up on 32-bit Windows apps and their memory limits. I don't recall if any 2 GB on the machine is accessible to a 32-bit or if it's a preset allocation in a specific area all 32-bit apps must share, including with other running 64-bit apps. If it's something like the latter, then that could explain why no more memory can be allocated if other apps are consuming that 2 GB address space only your 32-bit app can use.
Did you try the /3GB switch?
If someone on your team is familiar with memory profiling, that would be a good way to identify the root cause.
-
Hi Alex, thanks for the response!
I haven't tried the /3GB switch (I hadn't, in fact, ever heard of it. I'm not really a Windows guy by default. But I've just gone and read up about it...) Just to be clear: because the error I'm receiving is a client library exception, I should experiment with the /3GB boot option on the application / client server? This is not a message somehow being passed through from the database server, and I should instead be experimenting there? I'm a bit... wary of messing about with our production database servers, not to mention that rebooting them is not something we do lightly.
-
It's extremely likely it's a client side issue. If it was a DB issue, you generally see an error with a code like ORA-12345 or another set of numbers.
The /3GB switch cuts your kernel memory allocation from 2 GB to 1 GB, while giving your use mode app the extra GB. It is certainly possible that you solve one problem, but create others if your kernel comes under memory pressure.
It would be best to try this out on a test setup if you can ever reproduce the problem there to verify /3GB would fix your problem without negative side effects.
A possible contributing factor is how the VM manages and allocates memory.