Skip to content

ConvertFrom-JSON high memory consumption #7698

Description

@strawgate

Steps to reproduce

$JSON = @"
{
  "configGlossary:installationAt": "Philadelphia, PA",
  "configGlossary:adminEmail": "[email protected]",
  "configGlossary:poweredBy": "Cofax",
  "configGlossary:poweredByIcon": "\/images\/cofax.gif",
  "configGlossary:staticPath": "\/content\/static",
  "templateProcessorClass": "org.cofax.WysiwygTemplate",
  "templateLoaderClass": "org.cofax.FilesTemplateLoader",
  "templatePath": "templates",
  "templateOverridePath": "",
  "defaultListTemplate": "listTemplate.htm",
  "defaultFileTemplate": "articleTemplate.htm",
  "useJSP": false,
  "jspListTemplate": "listTemplate.jsp",
  "jspFileTemplate": "articleTemplate.jsp",
  "cachePackageTagsTrack": 200,
  "cachePackageTagsStore": 200,
  "cachePackageTagsRefresh": 60,
  "cacheTemplatesTrack": 100,
  "cacheTemplatesStore": 50,
  "cacheTemplatesRefresh": 15,
  "cachePagesTrack": 200,
  "cachePagesStore": 100,
  "cachePagesRefresh": 10,
  "cachePagesDirtyRead": 10,
  "searchEngineListTemplate": "forSearchEnginesList.htm",
  "searchEngineFileTemplate": "forSearchEngines.htm",
  "searchEngineRobotsDb": "WEB-INF\/robots.db",
  "useDataStore": true,
  "dataStoreClass": "org.cofax.SqlDataStore",
  "redirectionClass": "org.cofax.SqlRedirection",
  "dataStoreName": "cofax",
  "dataStoreDriver": "com.microsoft.jdbc.sqlserver.SQLServerDriver",
  "dataStoreUrl": "jdbc:microsoft:sqlserver:\/\/LOCALHOST:1433;DatabaseName=goon",
  "dataStoreUser": "sa",
  "dataStorePassword": "dataStoreTestQuery",
  "dataStoreTestQuery": "SET NOCOUNT ON;select test='test';",
  "dataStoreLogFile": "\/usr\/local\/tomcat\/logs\/datastore.log",
  "dataStoreInitConns": 10,
  "dataStoreMaxConns": 100,
  "dataStoreConnUsageLimit": 100,
  "dataStoreLogLevel": "debug",
  "maxUrlLength": 500
}
"@

$Count = 39999
$jsonarr = "[" + (($json + ",") * $Count) + $json + "]"
[system.gc]::Collect()
write-host ""
write-host ""
write-host "JSON -- Objects: $($Count + 1)"
write-host "JSON -- Length:  $($jsonarr.length / 1024 / 1024 ) Megabytes"

$BeforeMemory = $((get-process -pid $PID).WorkingSet)
[void] (convertfrom-json $jsonarr)
$AfterMemory = $((get-process -pid $PID).WorkingSet)

$MemoryDiff = $AfterMemory - $BeforeMemory

write-host "Memory Usage -- Before:     $($BeforeMemory  / 1024 / 1024) Megabytes"
write-host "Memory Usage -- After:      $($AfterMemory  / 1024 / 1024) Megabytes"
write-host "Memory Usage -- Difference: $($MemoryDiff  / 1024 / 1024) Megabytes"
write-host "Memory Usage -- Multiple:   $($MemoryDiff / $jsonarr.length)"

output:

JSON -- Objects: 40000
JSON -- Length:  67.4819955825806 Megabytes
Memory Usage -- Before:     210.8359375 Megabytes
Memory Usage -- After:      1240.90625 Megabytes
Memory Usage -- Difference: 1030.0703125 Megabytes
Memory Usage -- Multiple:   15.2643724241892

Actual behavior

Powershell Core ends up with 1 GB of memory used and peaks to 1.6 GB of memory utilization during convertfrom-json

Expected behavior

Powershell Core shouldn't need 1.6 gigabytes of memory to deserialize a 65MB JSON file.

Environment data

Name                           Value
----                           -----
PSVersion                      6.1.0-preview.3
PSEdition                      Core
GitCommitId                    v6.1.0-preview.3
OS                             Microsoft Windows 10.0.17134
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Issue-Enhancementthe issue is more of a feature request than a bugResolution-No ActivityIssue has had no activity for 6 months or moreWG-Cmdletsgeneral cmdlet issuesWG-Engine-Performancecore PowerShell engine, interpreter, and runtime performance

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions