Google Git
Sign in
cobalt / cobalt / da5c4f09a295755b8923113581bebdc12153e6dd / . / src / third_party / mozjs-45 / js / src / jsgc.cpp
blob: acc2db5384b9d76d0e3cf89309ed20da56bab7f2 [file] [log] [blame]
/* -*- Mode: C++; tab-width: 8; indent-tabs-mode: nil; c-basic-offset: 4 -*-
* vim: set ts=8 sts=4 et sw=4 tw=99:
* This Source Code Form is subject to the terms of the Mozilla Public
* License, v. 2.0. If a copy of the MPL was not distributed with this
* file, You can obtain one at http://mozilla.org/MPL/2.0/. */
/*
* This code implements an incremental mark-and-sweep garbage collector, with
* most sweeping carried out in the background on a parallel thread.
*
* Full vs. zone GC
* ----------------
*
* The collector can collect all zones at once, or a subset. These types of
* collection are referred to as a full GC and a zone GC respectively.
*
* The atoms zone is only collected in a full GC since objects in any zone may
* have pointers to atoms, and these are not recorded in the cross compartment
* pointer map. Also, the atoms zone is not collected if any thread has an
* AutoKeepAtoms instance on the stack, or there are any exclusive threads using
* the runtime.
*
* It is possible for an incremental collection that started out as a full GC to
* become a zone GC if new zones are created during the course of the
* collection.
*
* Incremental collection
* ----------------------
*
* For a collection to be carried out incrementally the following conditions
* must be met:
* - the collection must be run by calling js::GCSlice() rather than js::GC()
* - the GC mode must have been set to JSGC_MODE_INCREMENTAL with
* JS_SetGCParameter()
* - no thread may have an AutoKeepAtoms instance on the stack
*
* The last condition is an engine-internal mechanism to ensure that incremental
* collection is not carried out without the correct barriers being implemented.
* For more information see 'Incremental marking' below.
*
* If the collection is not incremental, all foreground activity happens inside
* a single call to GC() or GCSlice(). However the collection is not complete
* until the background sweeping activity has finished.
*
* An incremental collection proceeds as a series of slices, interleaved with
* mutator activity, i.e. running JavaScript code. Slices are limited by a time
* budget. The slice finishes as soon as possible after the requested time has
* passed.
*
* Collector states
* ----------------
*
* The collector proceeds through the following states, the current state being
* held in JSRuntime::gcIncrementalState:
*
* - MARK_ROOTS - marks the stack and other roots
* - MARK - incrementally marks reachable things
* - SWEEP - sweeps zones in groups and continues marking unswept zones
*
* The MARK_ROOTS activity always takes place in the first slice. The next two
* states can take place over one or more slices.
*
* In other words an incremental collection proceeds like this:
*
* Slice 1: MARK_ROOTS: Roots pushed onto the mark stack.
* MARK: The mark stack is processed by popping an element,
* marking it, and pushing its children.
*
* ... JS code runs ...
*
* Slice 2: MARK: More mark stack processing.
*
* ... JS code runs ...
*
* Slice n-1: MARK: More mark stack processing.
*
* ... JS code runs ...
*
* Slice n: MARK: Mark stack is completely drained.
* SWEEP: Select first group of zones to sweep and sweep them.
*
* ... JS code runs ...
*
* Slice n+1: SWEEP: Mark objects in unswept zones that were newly
* identified as alive (see below). Then sweep more zone
* groups.
*
* ... JS code runs ...
*
* Slice n+2: SWEEP: Mark objects in unswept zones that were newly
* identified as alive. Then sweep more zone groups.
*
* ... JS code runs ...
*
* Slice m: SWEEP: Sweeping is finished, and background sweeping
* started on the helper thread.
*
* ... JS code runs, remaining sweeping done on background thread ...
*
* When background sweeping finishes the GC is complete.
*
* Incremental marking
* -------------------
*
* Incremental collection requires close collaboration with the mutator (i.e.,
* JS code) to guarantee correctness.
*
* - During an incremental GC, if a memory location (except a root) is written
* to, then the value it previously held must be marked. Write barriers
* ensure this.
*
* - Any object that is allocated during incremental GC must start out marked.
*
* - Roots are marked in the first slice and hence don't need write barriers.
* Roots are things like the C stack and the VM stack.
*
* The problem that write barriers solve is that between slices the mutator can
* change the object graph. We must ensure that it cannot do this in such a way
* that makes us fail to mark a reachable object (marking an unreachable object
* is tolerable).
*
* We use a snapshot-at-the-beginning algorithm to do this. This means that we
* promise to mark at least everything that is reachable at the beginning of
* collection. To implement it we mark the old contents of every non-root memory
* location written to by the mutator while the collection is in progress, using
* write barriers. This is described in gc/Barrier.h.
*
* Incremental sweeping
* --------------------
*
* Sweeping is difficult to do incrementally because object finalizers must be
* run at the start of sweeping, before any mutator code runs. The reason is
* that some objects use their finalizers to remove themselves from caches. If
* mutator code was allowed to run after the start of sweeping, it could observe
* the state of the cache and create a new reference to an object that was just
* about to be destroyed.
*
* Sweeping all finalizable objects in one go would introduce long pauses, so
* instead sweeping broken up into groups of zones. Zones which are not yet
* being swept are still marked, so the issue above does not apply.
*
* The order of sweeping is restricted by cross compartment pointers - for
* example say that object |a| from zone A points to object |b| in zone B and
* neither object was marked when we transitioned to the SWEEP phase. Imagine we
* sweep B first and then return to the mutator. It's possible that the mutator
* could cause |a| to become alive through a read barrier (perhaps it was a
* shape that was accessed via a shape table). Then we would need to mark |b|,
* which |a| points to, but |b| has already been swept.
*
* So if there is such a pointer then marking of zone B must not finish before
* marking of zone A. Pointers which form a cycle between zones therefore
* restrict those zones to being swept at the same time, and these are found
* using Tarjan's algorithm for finding the strongly connected components of a
* graph.
*
* GC things without finalizers, and things with finalizers that are able to run
* in the background, are swept on the background thread. This accounts for most
* of the sweeping work.
*
* Reset
* -----
*
* During incremental collection it is possible, although unlikely, for
* conditions to change such that incremental collection is no longer safe. In
* this case, the collection is 'reset' by ResetIncrementalGC(). If we are in
* the mark state, this just stops marking, but if we have started sweeping
* already, we continue until we have swept the current zone group. Following a
* reset, a new non-incremental collection is started.
*
* Compacting GC
* -------------
*
* Compacting GC happens at the end of a major GC as part of the last slice.
* There are three parts:
*
* - Arenas are selected for compaction.
* - The contents of those arenas are moved to new arenas.
* - All references to moved things are updated.
*/
#include "jsgcinlines.h"
#include "mozilla/ArrayUtils.h"
#include "mozilla/DebugOnly.h"
#include "mozilla/MacroForEach.h"
#include "mozilla/MemoryReporting.h"
#include "mozilla/Move.h"
#include <ctype.h>
#include <string.h>
#if defined(STARBOARD)
#elif !defined(XP_WIN)
# include <sys/mman.h>
# include <unistd.h>
#endif
#include "jsapi.h"
#include "jsatom.h"
#include "jscntxt.h"
#include "jscompartment.h"
#include "jsfriendapi.h"
#include "jsobj.h"
#include "jsprf.h"
#include "jsscript.h"
#include "jstypes.h"
#include "jsutil.h"
#include "jswatchpoint.h"
#include "jsweakmap.h"
#ifdef XP_WIN
# include "jswin.h"
#endif
#include "gc/FindSCCs.h"
#include "gc/GCInternals.h"
#include "gc/GCTrace.h"
#include "gc/Marking.h"
#include "gc/Memory.h"
#include "jit/BaselineJIT.h"
#include "jit/IonCode.h"
#include "jit/JitcodeMap.h"
#include "js/SliceBudget.h"
#include "proxy/DeadObjectProxy.h"
#include "vm/Debugger.h"
#include "vm/ProxyObject.h"
#include "vm/Shape.h"
#include "vm/String.h"
#include "vm/Symbol.h"
#include "vm/Time.h"
#include "vm/TraceLogging.h"
#include "vm/WrapperObject.h"
#include "jsobjinlines.h"
#include "jsscriptinlines.h"
#include "vm/Stack-inl.h"
#include "vm/String-inl.h"
using namespace js;
using namespace js::gc;
using mozilla::ArrayLength;
using mozilla::Maybe;
using mozilla::Swap;
using JS::AutoGCRooter;
/* Perform a Full GC every 20 seconds if MaybeGC is called */
static const uint64_t GC_IDLE_FULL_SPAN = 20 * 1000 * 1000;
/* Increase the IGC marking slice time if we are in highFrequencyGC mode. */
static const int IGC_MARK_SLICE_MULTIPLIER = 2;
const AllocKind gc::slotsToThingKind[] = {
/* 0 */ AllocKind::OBJECT0, AllocKind::OBJECT2, AllocKind::OBJECT2, AllocKind::OBJECT4,
/* 4 */ AllocKind::OBJECT4, AllocKind::OBJECT8, AllocKind::OBJECT8, AllocKind::OBJECT8,
/* 8 */ AllocKind::OBJECT8, AllocKind::OBJECT12, AllocKind::OBJECT12, AllocKind::OBJECT12,
/* 12 */ AllocKind::OBJECT12, AllocKind::OBJECT16, AllocKind::OBJECT16, AllocKind::OBJECT16,
/* 16 */ AllocKind::OBJECT16
};
static_assert(JS_ARRAY_LENGTH(slotsToThingKind) == SLOTS_TO_THING_KIND_LIMIT,
"We have defined a slot count for each kind.");
// Assert that SortedArenaList::MinThingSize is <= the real minimum thing size.
#define CHECK_MIN_THING_SIZE_INNER(x_) \
static_assert(x_ >= SortedArenaList::MinThingSize, \
#x_ " is less than SortedArenaList::MinThingSize!");
#define CHECK_MIN_THING_SIZE(...) { __VA_ARGS__ }; /* Define the array. */ \
MOZ_FOR_EACH(CHECK_MIN_THING_SIZE_INNER, (), (__VA_ARGS__ UINT32_MAX))
const uint32_t Arena::ThingSizes[] = CHECK_MIN_THING_SIZE(
sizeof(JSFunction), /* AllocKind::FUNCTION */
sizeof(FunctionExtended), /* AllocKind::FUNCTION_EXTENDED */
sizeof(JSObject_Slots0), /* AllocKind::OBJECT0 */
sizeof(JSObject_Slots0), /* AllocKind::OBJECT0_BACKGROUND */
sizeof(JSObject_Slots2), /* AllocKind::OBJECT2 */
sizeof(JSObject_Slots2), /* AllocKind::OBJECT2_BACKGROUND */
sizeof(JSObject_Slots4), /* AllocKind::OBJECT4 */
sizeof(JSObject_Slots4), /* AllocKind::OBJECT4_BACKGROUND */
sizeof(JSObject_Slots8), /* AllocKind::OBJECT8 */
sizeof(JSObject_Slots8), /* AllocKind::OBJECT8_BACKGROUND */
sizeof(JSObject_Slots12), /* AllocKind::OBJECT12 */
sizeof(JSObject_Slots12), /* AllocKind::OBJECT12_BACKGROUND */
sizeof(JSObject_Slots16), /* AllocKind::OBJECT16 */
sizeof(JSObject_Slots16), /* AllocKind::OBJECT16_BACKGROUND */
sizeof(JSScript), /* AllocKind::SCRIPT */
sizeof(LazyScript), /* AllocKind::LAZY_SCRIPT */
sizeof(Shape), /* AllocKind::SHAPE */
sizeof(AccessorShape), /* AllocKind::ACCESSOR_SHAPE */
sizeof(BaseShape), /* AllocKind::BASE_SHAPE */
sizeof(ObjectGroup), /* AllocKind::OBJECT_GROUP */
sizeof(JSFatInlineString), /* AllocKind::FAT_INLINE_STRING */
sizeof(JSString), /* AllocKind::STRING */
sizeof(JSExternalString), /* AllocKind::EXTERNAL_STRING */
sizeof(JS::Symbol), /* AllocKind::SYMBOL */
sizeof(jit::JitCode), /* AllocKind::JITCODE */
);
#undef CHECK_MIN_THING_SIZE_INNER
#undef CHECK_MIN_THING_SIZE
#define OFFSET(type) uint32_t(sizeof(ArenaHeader) + (ArenaSize - sizeof(ArenaHeader)) % sizeof(type))
const uint32_t Arena::FirstThingOffsets[] = {
OFFSET(JSFunction), /* AllocKind::FUNCTION */
OFFSET(FunctionExtended), /* AllocKind::FUNCTION_EXTENDED */
OFFSET(JSObject_Slots0), /* AllocKind::OBJECT0 */
OFFSET(JSObject_Slots0), /* AllocKind::OBJECT0_BACKGROUND */
OFFSET(JSObject_Slots2), /* AllocKind::OBJECT2 */
OFFSET(JSObject_Slots2), /* AllocKind::OBJECT2_BACKGROUND */
OFFSET(JSObject_Slots4), /* AllocKind::OBJECT4 */
OFFSET(JSObject_Slots4), /* AllocKind::OBJECT4_BACKGROUND */
OFFSET(JSObject_Slots8), /* AllocKind::OBJECT8 */
OFFSET(JSObject_Slots8), /* AllocKind::OBJECT8_BACKGROUND */
OFFSET(JSObject_Slots12), /* AllocKind::OBJECT12 */
OFFSET(JSObject_Slots12), /* AllocKind::OBJECT12_BACKGROUND */
OFFSET(JSObject_Slots16), /* AllocKind::OBJECT16 */
OFFSET(JSObject_Slots16), /* AllocKind::OBJECT16_BACKGROUND */
OFFSET(JSScript), /* AllocKind::SCRIPT */
OFFSET(LazyScript), /* AllocKind::LAZY_SCRIPT */
OFFSET(Shape), /* AllocKind::SHAPE */
OFFSET(AccessorShape), /* AllocKind::ACCESSOR_SHAPE */
OFFSET(BaseShape), /* AllocKind::BASE_SHAPE */
OFFSET(ObjectGroup), /* AllocKind::OBJECT_GROUP */
OFFSET(JSFatInlineString), /* AllocKind::FAT_INLINE_STRING */
OFFSET(JSString), /* AllocKind::STRING */
OFFSET(JSExternalString), /* AllocKind::EXTERNAL_STRING */
OFFSET(JS::Symbol), /* AllocKind::SYMBOL */
OFFSET(jit::JitCode), /* AllocKind::JITCODE */
};
#undef OFFSET
struct js::gc::FinalizePhase
{
size_t length;
const AllocKind* kinds;
gcstats::Phase statsPhase;
};
#define PHASE(x, p) { ArrayLength(x), x, p }
/*
* Finalization order for incrementally swept things.
*/
static const AllocKind IncrementalPhaseStrings[] = {
AllocKind::EXTERNAL_STRING
};
static const AllocKind IncrementalPhaseScripts[] = {
AllocKind::SCRIPT,
AllocKind::LAZY_SCRIPT
};
static const AllocKind IncrementalPhaseJitCode[] = {
AllocKind::JITCODE
};
static const FinalizePhase IncrementalFinalizePhases[] = {
PHASE(IncrementalPhaseStrings, gcstats::PHASE_SWEEP_STRING),
PHASE(IncrementalPhaseScripts, gcstats::PHASE_SWEEP_SCRIPT),
PHASE(IncrementalPhaseJitCode, gcstats::PHASE_SWEEP_JITCODE)
};
/*
* Finalization order for things swept in the background.
*/
static const AllocKind BackgroundPhaseObjects[] = {
AllocKind::FUNCTION,
AllocKind::FUNCTION_EXTENDED,
AllocKind::OBJECT0_BACKGROUND,
AllocKind::OBJECT2_BACKGROUND,
AllocKind::OBJECT4_BACKGROUND,
AllocKind::OBJECT8_BACKGROUND,
AllocKind::OBJECT12_BACKGROUND,
AllocKind::OBJECT16_BACKGROUND
};
static const AllocKind BackgroundPhaseStringsAndSymbols[] = {
AllocKind::FAT_INLINE_STRING,
AllocKind::STRING,
AllocKind::SYMBOL
};
static const AllocKind BackgroundPhaseShapes[] = {
AllocKind::SHAPE,
AllocKind::ACCESSOR_SHAPE,
AllocKind::BASE_SHAPE,
AllocKind::OBJECT_GROUP
};
static const FinalizePhase BackgroundFinalizePhases[] = {
PHASE(BackgroundPhaseObjects, gcstats::PHASE_SWEEP_OBJECT),
PHASE(BackgroundPhaseStringsAndSymbols, gcstats::PHASE_SWEEP_STRING),
PHASE(BackgroundPhaseShapes, gcstats::PHASE_SWEEP_SHAPE)
};
#undef PHASE
template<>
JSObject*
ArenaCellIterImpl::get<JSObject>() const
{
MOZ_ASSERT(!done());
return reinterpret_cast<JSObject*>(getCell());
}
#ifdef DEBUG
void
ArenaHeader::checkSynchronizedWithFreeList() const
{
/*
* Do not allow to access the free list when its real head is still stored
* in FreeLists and is not synchronized with this one.
*/
MOZ_ASSERT(allocated());
/*
* We can be called from the background finalization thread when the free
* list in the zone can mutate at any moment. We cannot do any
* checks in this case.
*/
if (IsBackgroundFinalized(getAllocKind()) && zone->runtimeFromAnyThread()->gc.onBackgroundThread())
return;
FreeSpan firstSpan = firstFreeSpan.decompact(arenaAddress());
if (firstSpan.isEmpty())
return;
const FreeList* freeList = zone->arenas.getFreeList(getAllocKind());
if (freeList->isEmpty() || firstSpan.arenaAddress() != freeList->arenaAddress())
return;
/*
* Here this arena has free things, FreeList::lists[thingKind] is not
* empty and also points to this arena. Thus they must be the same.
*/
MOZ_ASSERT(freeList->isSameNonEmptySpan(firstSpan));
}
#endif
void
ArenaHeader::unmarkAll()
{
uintptr_t* word = chunk()->bitmap.arenaBits(this);
memset(word, 0, ArenaBitmapWords * sizeof(uintptr_t));
}
/* static */ void
Arena::staticAsserts()
{
static_assert(JS_ARRAY_LENGTH(ThingSizes) == size_t(AllocKind::LIMIT),
"We haven't defined all thing sizes.");
static_assert(JS_ARRAY_LENGTH(FirstThingOffsets) == size_t(AllocKind::LIMIT),
"We haven't defined all offsets.");
}
void
Arena::setAsFullyUnused(AllocKind thingKind)
{
FreeSpan fullSpan;
size_t thingSize = Arena::thingSize(thingKind);
fullSpan.initFinal(thingsStart(thingKind), thingsEnd() - thingSize, thingSize);
aheader.setFirstFreeSpan(&fullSpan);
}
template<typename T>
inline size_t
Arena::finalize(FreeOp* fop, AllocKind thingKind, size_t thingSize)
{
/* Enforce requirements on size of T. */
MOZ_ASSERT(thingSize % CellSize == 0);
MOZ_ASSERT(thingSize <= 255);
MOZ_ASSERT(aheader.allocated());
MOZ_ASSERT(thingKind == aheader.getAllocKind());
MOZ_ASSERT(thingSize == aheader.getThingSize());
MOZ_ASSERT(!aheader.hasDelayedMarking);
MOZ_ASSERT(!aheader.markOverflow);
MOZ_ASSERT(!aheader.allocatedDuringIncremental);
uintptr_t firstThing = thingsStart(thingKind);
uintptr_t firstThingOrSuccessorOfLastMarkedThing = firstThing;
uintptr_t lastThing = thingsEnd() - thingSize;
FreeSpan newListHead;
FreeSpan* newListTail = &newListHead;
size_t nmarked = 0;
if (MOZ_UNLIKELY(MemProfiler::enabled())) {
for (ArenaCellIterUnderFinalize i(&aheader); !i.done(); i.next()) {
T* t = i.get<T>();
if (t->asTenured().isMarked())
MemProfiler::MarkTenured(reinterpret_cast<void*>(t));
}
}
for (ArenaCellIterUnderFinalize i(&aheader); !i.done(); i.next()) {
T* t = i.get<T>();
if (t->asTenured().isMarked()) {
uintptr_t thing = reinterpret_cast<uintptr_t>(t);
if (thing != firstThingOrSuccessorOfLastMarkedThing) {
// We just finished passing over one or more free things,
// so record a new FreeSpan.
newListTail->initBoundsUnchecked(firstThingOrSuccessorOfLastMarkedThing,
thing - thingSize);
newListTail = newListTail->nextSpanUnchecked();
}
firstThingOrSuccessorOfLastMarkedThing = thing + thingSize;
nmarked++;
} else {
t->finalize(fop);
JS_POISON(t, JS_SWEPT_TENURED_PATTERN, thingSize);
TraceTenuredFinalize(t);
}
}
if (nmarked == 0) {
// Do nothing. The caller will update the arena header appropriately.
MOZ_ASSERT(newListTail == &newListHead);
JS_EXTRA_POISON(data, JS_SWEPT_TENURED_PATTERN, sizeof(data));
return nmarked;
}
MOZ_ASSERT(firstThingOrSuccessorOfLastMarkedThing != firstThing);
uintptr_t lastMarkedThing = firstThingOrSuccessorOfLastMarkedThing - thingSize;
if (lastThing == lastMarkedThing) {
// If the last thing was marked, we will have already set the bounds of
// the final span, and we just need to terminate the list.
newListTail->initAsEmpty();
} else {
// Otherwise, end the list with a span that covers the final stretch of free things.
newListTail->initFinal(firstThingOrSuccessorOfLastMarkedThing, lastThing, thingSize);
}
#ifdef DEBUG
size_t nfree = 0;
for (const FreeSpan* span = &newListHead; !span->isEmpty(); span = span->nextSpan())
nfree += span->length(thingSize);
MOZ_ASSERT(nfree + nmarked == thingsPerArena(thingSize));
#endif
aheader.setFirstFreeSpan(&newListHead);
return nmarked;
}
// Finalize arenas from src list, releasing empty arenas if keepArenas wasn't
// specified and inserting the others into the appropriate destination size
// bins.
template<typename T>
static inline bool
FinalizeTypedArenas(FreeOp* fop,
ArenaHeader** src,
SortedArenaList& dest,
AllocKind thingKind,
SliceBudget& budget,
ArenaLists::KeepArenasEnum keepArenas)
{
// When operating in the foreground, take the lock at the top.
Maybe<AutoLockGC> maybeLock;
if (!fop->onBackgroundThread())
maybeLock.emplace(fop->runtime());
// During background sweeping free arenas are released later on in
// sweepBackgroundThings().
MOZ_ASSERT_IF(fop->onBackgroundThread(), keepArenas == ArenaLists::KEEP_ARENAS);
size_t thingSize = Arena::thingSize(thingKind);
size_t thingsPerArena = Arena::thingsPerArena(thingSize);
while (ArenaHeader* aheader = *src) {
*src = aheader->next;
size_t nmarked = aheader->getArena()->finalize<T>(fop, thingKind, thingSize);
size_t nfree = thingsPerArena - nmarked;
if (nmarked)
dest.insertAt(aheader, nfree);
else if (keepArenas == ArenaLists::KEEP_ARENAS)
aheader->chunk()->recycleArena(aheader, dest, thingKind, thingsPerArena);
else
fop->runtime()->gc.releaseArena(aheader, maybeLock.ref());
budget.step(thingsPerArena);
if (budget.isOverBudget())
return false;
}
return true;
}
/*
* Finalize the list. On return, |al|'s cursor points to the first non-empty
* arena in the list (which may be null if all arenas are full).
*/
static bool
FinalizeArenas(FreeOp* fop,
ArenaHeader** src,
SortedArenaList& dest,
AllocKind thingKind,
SliceBudget& budget,
ArenaLists::KeepArenasEnum keepArenas)
{
switch (thingKind) {
case AllocKind::FUNCTION:
case AllocKind::FUNCTION_EXTENDED:
case AllocKind::OBJECT0:
case AllocKind::OBJECT0_BACKGROUND:
case AllocKind::OBJECT2:
case AllocKind::OBJECT2_BACKGROUND:
case AllocKind::OBJECT4:
case AllocKind::OBJECT4_BACKGROUND:
case AllocKind::OBJECT8:
case AllocKind::OBJECT8_BACKGROUND:
case AllocKind::OBJECT12:
case AllocKind::OBJECT12_BACKGROUND:
case AllocKind::OBJECT16:
case AllocKind::OBJECT16_BACKGROUND:
return FinalizeTypedArenas<JSObject>(fop, src, dest, thingKind, budget, keepArenas);
case AllocKind::SCRIPT:
return FinalizeTypedArenas<JSScript>(fop, src, dest, thingKind, budget, keepArenas);
case AllocKind::LAZY_SCRIPT:
return FinalizeTypedArenas<LazyScript>(fop, src, dest, thingKind, budget, keepArenas);
case AllocKind::SHAPE:
return FinalizeTypedArenas<Shape>(fop, src, dest, thingKind, budget, keepArenas);
case AllocKind::ACCESSOR_SHAPE:
return FinalizeTypedArenas<AccessorShape>(fop, src, dest, thingKind, budget, keepArenas);
case AllocKind::BASE_SHAPE:
return FinalizeTypedArenas<BaseShape>(fop, src, dest, thingKind, budget, keepArenas);
case AllocKind::OBJECT_GROUP:
return FinalizeTypedArenas<ObjectGroup>(fop, src, dest, thingKind, budget, keepArenas);
case AllocKind::STRING:
return FinalizeTypedArenas<JSString>(fop, src, dest, thingKind, budget, keepArenas);
case AllocKind::FAT_INLINE_STRING:
return FinalizeTypedArenas<JSFatInlineString>(fop, src, dest, thingKind, budget, keepArenas);
case AllocKind::EXTERNAL_STRING:
return FinalizeTypedArenas<JSExternalString>(fop, src, dest, thingKind, budget, keepArenas);
case AllocKind::SYMBOL:
return FinalizeTypedArenas<JS::Symbol>(fop, src, dest, thingKind, budget, keepArenas);
case AllocKind::JITCODE:
return FinalizeTypedArenas<jit::JitCode>(fop, src, dest, thingKind, budget, keepArenas);
default:
MOZ_CRASH("Invalid alloc kind");
}
}
Chunk*
ChunkPool::pop()
{
MOZ_ASSERT(bool(head_) == bool(count_));
if (!count_)
return nullptr;
return remove(head_);
}
void
ChunkPool::push(Chunk* chunk)
{
MOZ_ASSERT(!chunk->info.next);
MOZ_ASSERT(!chunk->info.prev);
chunk->info.age = 0;
chunk->info.next = head_;
if (head_)
head_->info.prev = chunk;
head_ = chunk;
++count_;
MOZ_ASSERT(verify());
}
Chunk*
ChunkPool::remove(Chunk* chunk)
{
MOZ_ASSERT(count_ > 0);
MOZ_ASSERT(contains(chunk));
if (head_ == chunk)
head_ = chunk->info.next;
if (chunk->info.prev)
chunk->info.prev->info.next = chunk->info.next;
if (chunk->info.next)
chunk->info.next->info.prev = chunk->info.prev;
chunk->info.next = chunk->info.prev = nullptr;
--count_;
MOZ_ASSERT(verify());
return chunk;
}
#ifdef DEBUG
bool
ChunkPool::contains(Chunk* chunk) const
{
verify();
for (Chunk* cursor = head_; cursor; cursor = cursor->info.next) {
if (cursor == chunk)
return true;
}
return false;
}
bool
ChunkPool::verify() const
{
MOZ_ASSERT(bool(head_) == bool(count_));
uint32_t count = 0;
for (Chunk* cursor = head_; cursor; cursor = cursor->info.next, ++count) {
MOZ_ASSERT_IF(cursor->info.prev, cursor->info.prev->info.next == cursor);
MOZ_ASSERT_IF(cursor->info.next, cursor->info.next->info.prev == cursor);
}
MOZ_ASSERT(count_ == count);
return true;
}
#endif
void
ChunkPool::Iter::next()
{
MOZ_ASSERT(!done());
current_ = current_->info.next;
}
ChunkPool
GCRuntime::expireEmptyChunkPool(bool shrinkBuffers, const AutoLockGC& lock)
{
/*
* Return old empty chunks to the system while preserving the order of
* other chunks in the list. This way, if the GC runs several times
* without emptying the list, the older chunks will stay at the tail
* and are more likely to reach the max age.
*/
MOZ_ASSERT(emptyChunks(lock).verify());
ChunkPool expired;
unsigned freeChunkCount = 0;
for (ChunkPool::Iter iter(emptyChunks(lock)); !iter.done();) {
Chunk* chunk = iter.get();
iter.next();
MOZ_ASSERT(chunk->unused());
MOZ_ASSERT(!fullChunks(lock).contains(chunk));
MOZ_ASSERT(!availableChunks(lock).contains(chunk));
if (freeChunkCount >= tunables.maxEmptyChunkCount() ||
(freeChunkCount >= tunables.minEmptyChunkCount(lock) &&
(shrinkBuffers || chunk->info.age == MAX_EMPTY_CHUNK_AGE)))
{
emptyChunks(lock).remove(chunk);
prepareToFreeChunk(chunk->info);
expired.push(chunk);
} else {
/* Keep the chunk but increase its age. */
++freeChunkCount;
++chunk->info.age;
}
}
MOZ_ASSERT(expired.verify());
MOZ_ASSERT(emptyChunks(lock).verify());
MOZ_ASSERT(emptyChunks(lock).count() <= tunables.maxEmptyChunkCount());
MOZ_ASSERT_IF(shrinkBuffers, emptyChunks(lock).count() <= tunables.minEmptyChunkCount(lock));
return expired;
}
static void
FreeChunkPool(JSRuntime* rt, ChunkPool& pool)
{
for (ChunkPool::Iter iter(pool); !iter.done();) {
Chunk* chunk = iter.get();
iter.next();
pool.remove(chunk);
MOZ_ASSERT(!chunk->info.numArenasFreeCommitted);
UnmapPages(static_cast<void*>(chunk), ChunkSize);
}
MOZ_ASSERT(pool.count() == 0);
}
void
GCRuntime::freeEmptyChunks(JSRuntime* rt, const AutoLockGC& lock)
{
FreeChunkPool(rt, emptyChunks(lock));
}
/* static */ Chunk*
Chunk::allocate(JSRuntime* rt)
{
Chunk* chunk = static_cast<Chunk*>(MapAlignedPages(ChunkSize, ChunkSize));
if (!chunk)
return nullptr;
chunk->init(rt);
rt->gc.stats.count(gcstats::STAT_NEW_CHUNK);
return chunk;
}
inline void
GCRuntime::prepareToFreeChunk(ChunkInfo& info)
{
MOZ_ASSERT(numArenasFreeCommitted >= info.numArenasFreeCommitted);
numArenasFreeCommitted -= info.numArenasFreeCommitted;
stats.count(gcstats::STAT_DESTROY_CHUNK);
#ifdef DEBUG
/*
* Let FreeChunkPool detect a missing prepareToFreeChunk call before it
* frees chunk.
*/
info.numArenasFreeCommitted = 0;
#endif
}
void Chunk::decommitAllArenas(JSRuntime* rt)
{
decommittedArenas.clear(true);
MarkPagesUnused(&arenas[0], ArenasPerChunk * ArenaSize);
info.freeArenasHead = nullptr;
info.lastDecommittedArenaOffset = 0;
info.numArenasFree = ArenasPerChunk;
info.numArenasFreeCommitted = 0;
}
void
Chunk::init(JSRuntime* rt)
{
JS_POISON(this, JS_FRESH_TENURED_PATTERN, ChunkSize);
/*
* We clear the bitmap to guard against JS::GCThingIsMarkedGray being called
* on uninitialized data, which would happen before the first GC cycle.
*/
bitmap.clear();
/*
* Decommit the arenas. We do this after poisoning so that if the OS does
* not have to recycle the pages, we still get the benefit of poisoning.
*/
decommitAllArenas(rt);
/* Initialize the chunk info. */
info.init();
new (&info.trailer) ChunkTrailer(rt);
/* The rest of info fields are initialized in pickChunk. */
}
/*
* Search for and return the next decommitted Arena. Our goal is to keep
* lastDecommittedArenaOffset "close" to a free arena. We do this by setting
* it to the most recently freed arena when we free, and forcing it to
* the last alloc + 1 when we allocate.
*/
uint32_t
Chunk::findDecommittedArenaOffset()
{
/* Note: lastFreeArenaOffset can be past the end of the list. */
for (unsigned i = info.lastDecommittedArenaOffset; i < ArenasPerChunk; i++)
if (decommittedArenas.get(i))
return i;
for (unsigned i = 0; i < info.lastDecommittedArenaOffset; i++)
if (decommittedArenas.get(i))
return i;
MOZ_CRASH("No decommitted arenas found.");
}
ArenaHeader*
Chunk::fetchNextDecommittedArena()
{
MOZ_ASSERT(info.numArenasFreeCommitted == 0);
MOZ_ASSERT(info.numArenasFree > 0);
unsigned offset = findDecommittedArenaOffset();
info.lastDecommittedArenaOffset = offset + 1;
--info.numArenasFree;
decommittedArenas.unset(offset);
Arena* arena = &arenas[offset];
MarkPagesInUse(arena, ArenaSize);
arena->aheader.setAsNotAllocated();
return &arena->aheader;
}
inline void
GCRuntime::updateOnFreeArenaAlloc(const ChunkInfo& info)
{
MOZ_ASSERT(info.numArenasFreeCommitted <= numArenasFreeCommitted);
--numArenasFreeCommitted;
}
inline ArenaHeader*
Chunk::fetchNextFreeArena(JSRuntime* rt)
{
MOZ_ASSERT(info.numArenasFreeCommitted > 0);
MOZ_ASSERT(info.numArenasFreeCommitted <= info.numArenasFree);
ArenaHeader* aheader = info.freeArenasHead;
info.freeArenasHead = aheader->next;
--info.numArenasFreeCommitted;
--info.numArenasFree;
rt->gc.updateOnFreeArenaAlloc(info);
return aheader;
}
ArenaHeader*
Chunk::allocateArena(JSRuntime* rt, Zone* zone, AllocKind thingKind, const AutoLockGC& lock)
{
ArenaHeader* aheader = info.numArenasFreeCommitted > 0
? fetchNextFreeArena(rt)
: fetchNextDecommittedArena();
aheader->init(zone, thingKind);
updateChunkListAfterAlloc(rt, lock);
return aheader;
}
inline void
GCRuntime::updateOnArenaFree(const ChunkInfo& info)
{
++numArenasFreeCommitted;
}
void
Chunk::addArenaToFreeList(JSRuntime* rt, ArenaHeader* aheader)
{
MOZ_ASSERT(!aheader->allocated());
aheader->next = info.freeArenasHead;
info.freeArenasHead = aheader;
++info.numArenasFreeCommitted;
++info.numArenasFree;
rt->gc.updateOnArenaFree(info);
}
void
Chunk::addArenaToDecommittedList(JSRuntime* rt, const ArenaHeader* aheader)
{
++info.numArenasFree;
decommittedArenas.set(Chunk::arenaIndex(aheader->arenaAddress()));
}
void
Chunk::recycleArena(ArenaHeader* aheader, SortedArenaList& dest, AllocKind thingKind,
size_t thingsPerArena)
{
aheader->getArena()->setAsFullyUnused(thingKind);
dest.insertAt(aheader, thingsPerArena);
}
void
Chunk::releaseArena(JSRuntime* rt, ArenaHeader* aheader, const AutoLockGC& lock)
{
MOZ_ASSERT(aheader->allocated());
MOZ_ASSERT(!aheader->hasDelayedMarking);
aheader->setAsNotAllocated();
addArenaToFreeList(rt, aheader);
updateChunkListAfterFree(rt, lock);
}
bool
Chunk::decommitOneFreeArena(JSRuntime* rt, AutoLockGC& lock)
{
MOZ_ASSERT(info.numArenasFreeCommitted > 0);
ArenaHeader* aheader = fetchNextFreeArena(rt);
updateChunkListAfterAlloc(rt, lock);
bool ok;
{
AutoUnlockGC unlock(lock);
ok = MarkPagesUnused(aheader->getArena(), ArenaSize);
}
if (ok)
addArenaToDecommittedList(rt, aheader);
else
addArenaToFreeList(rt, aheader);
updateChunkListAfterFree(rt, lock);
return ok;
}
void
Chunk::decommitAllArenasWithoutUnlocking(const AutoLockGC& lock)
{
for (size_t i = 0; i < ArenasPerChunk; ++i) {
if (decommittedArenas.get(i) || arenas[i].aheader.allocated())
continue;
if (MarkPagesUnused(&arenas[i], ArenaSize)) {
info.numArenasFreeCommitted--;
decommittedArenas.set(i);
}
}
}
void
Chunk::updateChunkListAfterAlloc(JSRuntime* rt, const AutoLockGC& lock)
{
if (MOZ_UNLIKELY(!hasAvailableArenas())) {
rt->gc.availableChunks(lock).remove(this);
rt->gc.fullChunks(lock).push(this);
}
}
void
Chunk::updateChunkListAfterFree(JSRuntime* rt, const AutoLockGC& lock)
{
if (info.numArenasFree == 1) {
rt->gc.fullChunks(lock).remove(this);
rt->gc.availableChunks(lock).push(this);
} else if (!unused()) {
MOZ_ASSERT(!rt->gc.fullChunks(lock).contains(this));
MOZ_ASSERT(rt->gc.availableChunks(lock).contains(this));
MOZ_ASSERT(!rt->gc.emptyChunks(lock).contains(this));
} else {
MOZ_ASSERT(unused());
rt->gc.availableChunks(lock).remove(this);
decommitAllArenas(rt);
rt->gc.emptyChunks(lock).push(this);
}
}
inline bool
GCRuntime::wantBackgroundAllocation(const AutoLockGC& lock) const
{
// To minimize memory waste, we do not want to run the background chunk
// allocation if we already have some empty chunks or when the runtime has
// a small heap size (and therefore likely has a small growth rate).
return allocTask.enabled() &&
emptyChunks(lock).count() < tunables.minEmptyChunkCount(lock) &&
(fullChunks(lock).count() + availableChunks(lock).count()) >= 4;
}
void
GCRuntime::startBackgroundAllocTaskIfIdle()
{
AutoLockHelperThreadState helperLock;
if (allocTask.isRunning())
return;
// Join the previous invocation of the task. This will return immediately
// if the thread has never been started.
allocTask.joinWithLockHeld();
allocTask.startWithLockHeld();
}
Chunk*
GCRuntime::pickChunk(const AutoLockGC& lock,
AutoMaybeStartBackgroundAllocation& maybeStartBackgroundAllocation)
{
if (availableChunks(lock).count())
return availableChunks(lock).head();
Chunk* chunk = emptyChunks(lock).pop();
if (!chunk) {
chunk = Chunk::allocate(rt);
if (!chunk)
return nullptr;
MOZ_ASSERT(chunk->info.numArenasFreeCommitted == 0);
}
MOZ_ASSERT(chunk->unused());
MOZ_ASSERT(!fullChunks(lock).contains(chunk));
if (wantBackgroundAllocation(lock))
maybeStartBackgroundAllocation.tryToStartBackgroundAllocation(rt);
chunkAllocationSinceLastGC = true;
availableChunks(lock).push(chunk);
return chunk;
}
ArenaHeader*
GCRuntime::allocateArena(Chunk* chunk, Zone* zone, AllocKind thingKind, const AutoLockGC& lock)
{
MOZ_ASSERT(chunk->hasAvailableArenas());
// Fail the allocation if we are over our heap size limits.
if (!rt->isHeapMinorCollecting() &&
!isHeapCompacting() &&
usage.gcBytes() >= tunables.gcMaxBytes())
{
return nullptr;
}
ArenaHeader* aheader = chunk->allocateArena(rt, zone, thingKind, lock);
zone->usage.addGCArena();
// Trigger an incremental slice if needed.
if (!rt->isHeapMinorCollecting() && !isHeapCompacting())
maybeAllocTriggerZoneGC(zone, lock);
return aheader;
}
void
GCRuntime::releaseArena(ArenaHeader* aheader, const AutoLockGC& lock)
{
aheader->zone->usage.removeGCArena();
if (isBackgroundSweeping())
aheader->zone->threshold.updateForRemovedArena(tunables);
return aheader->chunk()->releaseArena(rt, aheader, lock);
}
GCRuntime::GCRuntime(JSRuntime* rt) :
rt(rt),
systemZone(nullptr),
nursery(rt),
storeBuffer(rt, nursery),
stats(rt),
marker(rt),
usage(nullptr),
mMemProfiler(rt),
maxMallocBytes(0),
nextCellUniqueId_(LargestTaggedNullCellPointer + 1), // Ensure disjoint from null tagged pointers.
numArenasFreeCommitted(0),
verifyPreData(nullptr),
chunkAllocationSinceLastGC(false),
nextFullGCTime(0),
lastGCTime(PRMJ_Now()),
mode(JSGC_MODE_INCREMENTAL),
numActiveZoneIters(0),
decommitThreshold(32 * 1024 * 1024),
cleanUpEverything(false),
grayBufferState(GCRuntime::GrayBufferState::Unused),
grayBitsValid(false),
majorGCTriggerReason(JS::gcreason::NO_REASON),
minorGCTriggerReason(JS::gcreason::NO_REASON),
fullGCForAtomsRequested_(false),
minorGCNumber(0),
majorGCNumber(0),
jitReleaseNumber(0),
number(0),
startNumber(0),
isFull(false),
#ifdef DEBUG
disableStrictProxyCheckingCount(0),
#endif
incrementalState(gc::NO_INCREMENTAL),
lastMarkSlice(false),
sweepOnBackgroundThread(false),
foundBlackGrayEdges(false),
freeLifoAlloc(JSRuntime::TEMP_LIFO_ALLOC_PRIMARY_CHUNK_SIZE),
zoneGroupIndex(0),
zoneGroups(nullptr),
currentZoneGroup(nullptr),
sweepZone(nullptr),
sweepKindIndex(0),
abortSweepAfterCurrentGroup(false),
arenasAllocatedDuringSweep(nullptr),
startedCompacting(false),
relocatedArenasToRelease(nullptr),
#ifdef JS_GC_MARKING_VALIDATION
markingValidator(nullptr),
#endif
interFrameGC(false),
defaultTimeBudget_(SliceBudget::UnlimitedTimeBudget),
incrementalAllowed(true),
generationalDisabled(0),
compactingEnabled(true),
compactingDisabledCount(0),
manipulatingDeadZones(false),
objectsMarkedInDeadZones(0),
poked(false),
#ifdef JS_GC_ZEAL
zealMode(0),
zealFrequency(0),
nextScheduled(0),
deterministicOnly(false),
incrementalLimit(0),
#endif
validate(true),
fullCompartmentChecks(false),
mallocBytesUntilGC(0),
mallocGCTriggered(false),
alwaysPreserveCode(false),
#ifdef DEBUG
inUnsafeRegion(0),
noGCOrAllocationCheck(0),
#endif
lock(nullptr),
allocTask(rt, emptyChunks_),
helperState(rt)
{
setGCMode(JSGC_MODE_GLOBAL);
}
#ifdef JS_GC_ZEAL
void
GCRuntime::getZeal(uint8_t* zeal, uint32_t* frequency, uint32_t* scheduled)
{
*zeal = zealMode;
*frequency = zealFrequency;
*scheduled = nextScheduled;
}
const char* gc::ZealModeHelpText =
" Specifies how zealous the garbage collector should be. Values for level:\n"
" 0: Normal amount of collection\n"
" 1: Collect when roots are added or removed\n"
" 2: Collect when every N allocations (default: 100)\n"
" 3: Collect when the window paints (browser only)\n"
" 4: Verify pre write barriers between instructions\n"
" 5: Verify pre write barriers between paints\n"
" 6: Verify stack rooting\n"
" 7: Collect the nursery every N nursery allocations\n"
" 8: Incremental GC in two slices: 1) mark roots 2) finish collection\n"
" 9: Incremental GC in two slices: 1) mark all 2) new marking and finish\n"
" 10: Incremental GC in multiple slices\n"
" 11: unused\n"
" 12: unused\n"
" 13: Check internal hashtables on minor GC\n"
" 14: Perform a shrinking collection every N allocations\n";
void
GCRuntime::setZeal(uint8_t zeal, uint32_t frequency)
{
if (verifyPreData)
VerifyBarriers(rt, PreBarrierVerifier);
if (zealMode == ZealGenerationalGCValue) {
evictNursery(JS::gcreason::DEBUG_GC);
nursery.leaveZealMode();
}
if (zeal == ZealGenerationalGCValue)
nursery.enterZealMode();
bool schedule = zeal >= js::gc::ZealAllocValue;
zealMode = zeal;
zealFrequency = frequency;
nextScheduled = schedule ? frequency : 0;
}
void
GCRuntime::setNextScheduled(uint32_t count)
{
nextScheduled = count;
}
bool
GCRuntime::parseAndSetZeal(const char* str)
{
int zeal = -1;
int frequency = -1;
if (isdigit(str[0])) {
zeal = atoi(str);
const char* p = strchr(str, ',');
if (!p)
frequency = JS_DEFAULT_ZEAL_FREQ;
else
frequency = atoi(p + 1);
}
if (zeal < 0 || zeal > ZealLimit || frequency <= 0) {
fprintf(stderr, "Format: JS_GC_ZEAL=level[,N]\n");
fputs(ZealModeHelpText, stderr);
return false;
}
setZeal(zeal, frequency);
return true;
}
#endif
/*
* Lifetime in number of major GCs for type sets attached to scripts containing
* observed types.
*/
static const uint64_t JIT_SCRIPT_RELEASE_TYPES_PERIOD = 20;
bool
GCRuntime::init(uint32_t maxbytes, uint32_t maxNurseryBytes)
{
InitMemorySubsystem();
lock = PR_NewLock();
if (!lock)
return false;
if (!rootsHash.init(256))
return false;
if (!helperState.init())
return false;
/*
* Separate gcMaxMallocBytes from gcMaxBytes but initialize to maxbytes
* for default backward API compatibility.
*/
AutoLockGC lock(rt);
tunables.setParameter(JSGC_MAX_BYTES, maxbytes, lock);
setMaxMallocBytes(maxbytes);
const char* size = js_sb_getenv("JSGC_MARK_STACK_LIMIT");
if (size)
setMarkStackLimit(atoi(size), lock);
jitReleaseNumber = majorGCNumber + JIT_SCRIPT_RELEASE_TYPES_PERIOD;
if (!nursery.init(maxNurseryBytes))
return false;
if (!nursery.isEnabled()) {
MOZ_ASSERT(nursery.nurserySize() == 0);
++rt->gc.generationalDisabled;
} else {
MOZ_ASSERT(nursery.nurserySize() > 0);
if (!storeBuffer.enable())
return false;
}
#ifdef JS_GC_ZEAL
const char* zealSpec = js_sb_getenv("JS_GC_ZEAL");
if (zealSpec && zealSpec[0] && !parseAndSetZeal(zealSpec))
return false;
#endif
if (!InitTrace(*this))
return false;
if (!marker.init(mode))
return false;
return true;
}
void
GCRuntime::finish()
{
/* Wait for the nursery sweeping to end. */
if (rt->gc.nursery.isEnabled())
rt->gc.nursery.waitBackgroundFreeEnd();
/*
* Wait until the background finalization and allocation stops and the
* helper thread shuts down before we forcefully release any remaining GC
* memory.
*/
helperState.finish();
allocTask.cancel(GCParallelTask::CancelAndWait);
#ifdef JS_GC_ZEAL
/* Free memory associated with GC verification. */
finishVerifier();
#endif
/* Delete all remaining zones. */
if (rt->gcInitialized) {
AutoSetThreadIsSweeping threadIsSweeping;
for (ZonesIter zone(rt, WithAtoms); !zone.done(); zone.next()) {
for (CompartmentsInZoneIter comp(zone); !comp.done(); comp.next())
js_delete(comp.get());
js_delete(zone.get());
}
}
zones.clear();
FreeChunkPool(rt, fullChunks_);
FreeChunkPool(rt, availableChunks_);
FreeChunkPool(rt, emptyChunks_);
if (lock) {
PR_DestroyLock(lock);
lock = nullptr;
}
FinishTrace();
}
template <typename T>
static void
FinishPersistentRootedChain(mozilla::LinkedList<PersistentRooted<T>>& list)
{
while (!list.isEmpty())
list.getFirst()->reset();
}
void
js::gc::FinishPersistentRootedChains(RootLists& roots)
{
FinishPersistentRootedChain(roots.getPersistentRootedList<JSObject*>());
FinishPersistentRootedChain(roots.getPersistentRootedList<JSScript*>());
FinishPersistentRootedChain(roots.getPersistentRootedList<JSString*>());
FinishPersistentRootedChain(roots.getPersistentRootedList<jsid>());
FinishPersistentRootedChain(roots.getPersistentRootedList<Value>());
FinishPersistentRootedChain(roots.heapRoots_[THING_ROOT_TRACEABLE]);
}
void
GCRuntime::finishRoots()
{
if (rootsHash.initialized())
rootsHash.clear();
FinishPersistentRootedChains(rt->mainThread.roots);
}
void
GCRuntime::setParameter(JSGCParamKey key, uint32_t value, AutoLockGC& lock)
{
switch (key) {
case JSGC_MAX_MALLOC_BYTES:
setMaxMallocBytes(value);
for (ZonesIter zone(rt, WithAtoms); !zone.done(); zone.next())
zone->setGCMaxMallocBytes(maxMallocBytesAllocated() * 0.9);
break;
case JSGC_SLICE_TIME_BUDGET:
defaultTimeBudget_ = value ? value : SliceBudget::UnlimitedTimeBudget;
break;
case JSGC_MARK_STACK_LIMIT:
setMarkStackLimit(value, lock);
break;
case JSGC_DECOMMIT_THRESHOLD:
decommitThreshold = value * 1024 * 1024;
break;
case JSGC_MODE:
mode = JSGCMode(value);
MOZ_ASSERT(mode == JSGC_MODE_GLOBAL ||
mode == JSGC_MODE_COMPARTMENT ||
mode == JSGC_MODE_INCREMENTAL);
break;
case JSGC_COMPACTING_ENABLED:
compactingEnabled = value != 0;
break;
default:
tunables.setParameter(key, value, lock);
for (ZonesIter zone(rt, WithAtoms); !zone.done(); zone.next()) {
zone->threshold.updateAfterGC(zone->usage.gcBytes(), GC_NORMAL, tunables,
schedulingState, lock);
}
}
}
void
GCSchedulingTunables::setParameter(JSGCParamKey key, uint32_t value, const AutoLockGC& lock)
{
switch(key) {
case JSGC_MAX_BYTES:
gcMaxBytes_ = value;
break;
case JSGC_HIGH_FREQUENCY_TIME_LIMIT:
highFrequencyThresholdUsec_ = value * PRMJ_USEC_PER_MSEC;
break;
case JSGC_HIGH_FREQUENCY_LOW_LIMIT:
highFrequencyLowLimitBytes_ = value * 1024 * 1024;
if (highFrequencyLowLimitBytes_ >= highFrequencyHighLimitBytes_)
highFrequencyHighLimitBytes_ = highFrequencyLowLimitBytes_ + 1;
MOZ_ASSERT(highFrequencyHighLimitBytes_ > highFrequencyLowLimitBytes_);
break;
case JSGC_HIGH_FREQUENCY_HIGH_LIMIT:
MOZ_ASSERT(value > 0);
highFrequencyHighLimitBytes_ = value * 1024 * 1024;
if (highFrequencyHighLimitBytes_ <= highFrequencyLowLimitBytes_)
highFrequencyLowLimitBytes_ = highFrequencyHighLimitBytes_ - 1;
MOZ_ASSERT(highFrequencyHighLimitBytes_ > highFrequencyLowLimitBytes_);
break;
case JSGC_HIGH_FREQUENCY_HEAP_GROWTH_MAX:
highFrequencyHeapGrowthMax_ = value / 100.0;
MOZ_ASSERT(highFrequencyHeapGrowthMax_ / 0.85 > 1.0);
break;
case JSGC_HIGH_FREQUENCY_HEAP_GROWTH_MIN:
highFrequencyHeapGrowthMin_ = value / 100.0;
MOZ_ASSERT(highFrequencyHeapGrowthMin_ / 0.85 > 1.0);
break;
case JSGC_LOW_FREQUENCY_HEAP_GROWTH:
lowFrequencyHeapGrowth_ = value / 100.0;
MOZ_ASSERT(lowFrequencyHeapGrowth_ / 0.9 > 1.0);
break;
case JSGC_DYNAMIC_HEAP_GROWTH:
dynamicHeapGrowthEnabled_ = value != 0;
break;
case JSGC_DYNAMIC_MARK_SLICE:
dynamicMarkSliceEnabled_ = value != 0;
break;
case JSGC_ALLOCATION_THRESHOLD:
gcZoneAllocThresholdBase_ = value * 1024 * 1024;
break;
case JSGC_MIN_EMPTY_CHUNK_COUNT:
minEmptyChunkCount_ = value;
if (minEmptyChunkCount_ > maxEmptyChunkCount_)
maxEmptyChunkCount_ = minEmptyChunkCount_;
MOZ_ASSERT(maxEmptyChunkCount_ >= minEmptyChunkCount_);
break;
case JSGC_MAX_EMPTY_CHUNK_COUNT:
maxEmptyChunkCount_ = value;
if (minEmptyChunkCount_ > maxEmptyChunkCount_)
minEmptyChunkCount_ = maxEmptyChunkCount_;
MOZ_ASSERT(maxEmptyChunkCount_ >= minEmptyChunkCount_);
break;
default:
MOZ_CRASH("Unknown GC parameter.");
}
}
uint32_t
GCRuntime::getParameter(JSGCParamKey key, const AutoLockGC& lock)
{
switch (key) {
case JSGC_MAX_BYTES:
return uint32_t(tunables.gcMaxBytes());
case JSGC_MAX_MALLOC_BYTES:
return maxMallocBytes;
case JSGC_BYTES:
return uint32_t(usage.gcBytes());
case JSGC_MODE:
return uint32_t(mode);
case JSGC_UNUSED_CHUNKS:
return uint32_t(emptyChunks(lock).count());
case JSGC_TOTAL_CHUNKS:
return uint32_t(fullChunks(lock).count() +
availableChunks(lock).count() +
emptyChunks(lock).count());
case JSGC_SLICE_TIME_BUDGET:
if (defaultTimeBudget_ == SliceBudget::UnlimitedTimeBudget) {
return 0;
} else {
MOZ_RELEASE_ASSERT(defaultTimeBudget_ >= 0);
MOZ_RELEASE_ASSERT(defaultTimeBudget_ < UINT32_MAX);
return uint32_t(defaultTimeBudget_);
}
case JSGC_MARK_STACK_LIMIT:
return marker.maxCapacity();
case JSGC_HIGH_FREQUENCY_TIME_LIMIT:
return tunables.highFrequencyThresholdUsec();
case JSGC_HIGH_FREQUENCY_LOW_LIMIT:
return tunables.highFrequencyLowLimitBytes() / 1024 / 1024;
case JSGC_HIGH_FREQUENCY_HIGH_LIMIT:
return tunables.highFrequencyHighLimitBytes() / 1024 / 1024;
case JSGC_HIGH_FREQUENCY_HEAP_GROWTH_MAX:
return uint32_t(tunables.highFrequencyHeapGrowthMax() * 100);
case JSGC_HIGH_FREQUENCY_HEAP_GROWTH_MIN:
return uint32_t(tunables.highFrequencyHeapGrowthMin() * 100);
case JSGC_LOW_FREQUENCY_HEAP_GROWTH:
return uint32_t(tunables.lowFrequencyHeapGrowth() * 100);
case JSGC_DYNAMIC_HEAP_GROWTH:
return tunables.isDynamicHeapGrowthEnabled();
case JSGC_DYNAMIC_MARK_SLICE:
return tunables.isDynamicMarkSliceEnabled();
case JSGC_ALLOCATION_THRESHOLD:
return tunables.gcZoneAllocThresholdBase() / 1024 / 1024;
case JSGC_MIN_EMPTY_CHUNK_COUNT:
return tunables.minEmptyChunkCount(lock);
case JSGC_MAX_EMPTY_CHUNK_COUNT:
return tunables.maxEmptyChunkCount();
case JSGC_COMPACTING_ENABLED:
return compactingEnabled;
default:
MOZ_ASSERT(key == JSGC_NUMBER);
return uint32_t(number);
}
}
void
GCRuntime::setMarkStackLimit(size_t limit, AutoLockGC& lock)
{
MOZ_ASSERT(!rt->isHeapBusy());
AutoUnlockGC unlock(lock);
AutoStopVerifyingBarriers pauseVerification(rt, false);
marker.setMaxCapacity(limit);
}
bool
GCRuntime::addBlackRootsTracer(JSTraceDataOp traceOp, void* data)
{
AssertHeapIsIdle(rt);
return !!blackRootTracers.append(Callback<JSTraceDataOp>(traceOp, data));
}
void
GCRuntime::removeBlackRootsTracer(JSTraceDataOp traceOp, void* data)
{
// Can be called from finalizers
for (size_t i = 0; i < blackRootTracers.length(); i++) {
Callback<JSTraceDataOp>* e = &blackRootTracers[i];
if (e->op == traceOp && e->data == data) {
blackRootTracers.erase(e);
}
}
}
void
GCRuntime::setGrayRootsTracer(JSTraceDataOp traceOp, void* data)
{
AssertHeapIsIdle(rt);
grayRootTracer.op = traceOp;
grayRootTracer.data = data;
}
void
GCRuntime::setGCCallback(JSGCCallback callback, void* data)
{
gcCallback.op = callback;
gcCallback.data = data;
}
void
GCRuntime::callGCCallback(JSGCStatus status) const
{
if (gcCallback.op)
gcCallback.op(rt, status, gcCallback.data);
}
namespace {
class AutoNotifyGCActivity {
public:
explicit AutoNotifyGCActivity(GCRuntime& gc) : gc_(gc) {
if (!gc_.isIncrementalGCInProgress()) {
gcstats::AutoPhase ap(gc_.stats, gcstats::PHASE_GC_BEGIN);
gc_.callGCCallback(JSGC_BEGIN);
}
}
~AutoNotifyGCActivity() {
if (!gc_.isIncrementalGCInProgress()) {
gcstats::AutoPhase ap(gc_.stats, gcstats::PHASE_GC_END);
gc_.callGCCallback(JSGC_END);
}
}
private:
GCRuntime& gc_;
};
} // (anon)
bool
GCRuntime::addFinalizeCallback(JSFinalizeCallback callback, void* data)
{
return finalizeCallbacks.append(Callback<JSFinalizeCallback>(callback, data));
}
void
GCRuntime::removeFinalizeCallback(JSFinalizeCallback callback)
{
for (Callback<JSFinalizeCallback>* p = finalizeCallbacks.begin();
p < finalizeCallbacks.end(); p++)
{
if (p->op == callback) {
finalizeCallbacks.erase(p);
break;
}
}
}
void
GCRuntime::callFinalizeCallbacks(FreeOp* fop, JSFinalizeStatus status) const
{
for (auto& p : finalizeCallbacks) {
p.op(fop, status, !isFull, p.data);
}
}
bool
GCRuntime::addWeakPointerZoneGroupCallback(JSWeakPointerZoneGroupCallback callback, void* data)
{
return updateWeakPointerZoneGroupCallbacks.append(
Callback<JSWeakPointerZoneGroupCallback>(callback, data));
}
void
GCRuntime::removeWeakPointerZoneGroupCallback(JSWeakPointerZoneGroupCallback callback)
{
for (auto& p : updateWeakPointerZoneGroupCallbacks) {
if (p.op == callback) {
updateWeakPointerZoneGroupCallbacks.erase(&p);
break;
}
}
}
void
GCRuntime::callWeakPointerZoneGroupCallbacks() const
{
for (auto const& p : updateWeakPointerZoneGroupCallbacks) {
p.op(rt, p.data);
}
}
bool
GCRuntime::addWeakPointerCompartmentCallback(JSWeakPointerCompartmentCallback callback, void* data)
{
return updateWeakPointerCompartmentCallbacks.append(
Callback<JSWeakPointerCompartmentCallback>(callback, data));
}
void
GCRuntime::removeWeakPointerCompartmentCallback(JSWeakPointerCompartmentCallback callback)
{
for (auto& p : updateWeakPointerCompartmentCallbacks) {
if (p.op == callback) {
updateWeakPointerCompartmentCallbacks.erase(&p);
break;
}
}
}
void
GCRuntime::callWeakPointerCompartmentCallbacks(JSCompartment* comp) const
{
for (auto const& p : updateWeakPointerCompartmentCallbacks) {
p.op(rt, comp, p.data);
}
}
JS::GCSliceCallback
GCRuntime::setSliceCallback(JS::GCSliceCallback callback) {
return stats.setSliceCallback(callback);
}
bool
GCRuntime::addRoot(Value* vp, const char* name)
{
/*
* Sometimes Firefox will hold weak references to objects and then convert
* them to strong references by calling AddRoot (e.g., via PreserveWrapper,
* or ModifyBusyCount in workers). We need a read barrier to cover these
* cases.
*/
if (isIncrementalGCInProgress())
HeapValue::writeBarrierPre(*vp);
return rootsHash.put(vp, name);
}
void
GCRuntime::removeRoot(Value* vp)
{
rootsHash.remove(vp);
poke();
}
extern JS_FRIEND_API(bool)
js::AddRawValueRoot(JSContext* cx, Value* vp, const char* name)
{
MOZ_ASSERT(vp);
MOZ_ASSERT(name);
bool ok = cx->runtime()->gc.addRoot(vp, name);
if (!ok)
JS_ReportOutOfMemory(cx);
return ok;
}
extern JS_FRIEND_API(void)
js::RemoveRawValueRoot(JSContext* cx, Value* vp)
{
cx->runtime()->gc.removeRoot(vp);
}
void
GCRuntime::setMaxMallocBytes(size_t value)
{
/*
* For compatibility treat any value that exceeds PTRDIFF_T_MAX to
* mean that value.
*/
maxMallocBytes = (ptrdiff_t(value) >= 0) ? value : size_t(-1) >> 1;
resetMallocBytes();
for (ZonesIter zone(rt, WithAtoms); !zone.done(); zone.next())
zone->setGCMaxMallocBytes(value);
}
void
GCRuntime::resetMallocBytes()
{
mallocBytesUntilGC = ptrdiff_t(maxMallocBytes);
mallocGCTriggered = false;
}
void
GCRuntime::updateMallocCounter(JS::Zone* zone, size_t nbytes)
{
mallocBytesUntilGC -= ptrdiff_t(nbytes);
if (MOZ_UNLIKELY(isTooMuchMalloc()))
onTooMuchMalloc();
else if (zone)
zone->updateMallocCounter(nbytes);
}
void
GCRuntime::onTooMuchMalloc()
{
if (!mallocGCTriggered)
mallocGCTriggered = triggerGC(JS::gcreason::TOO_MUCH_MALLOC);
}
double
ZoneHeapThreshold::allocTrigger(bool highFrequencyGC) const
{
return (highFrequencyGC ? 0.85 : 0.9) * gcTriggerBytes();
}
/* static */ double
ZoneHeapThreshold::computeZoneHeapGrowthFactorForHeapSize(size_t lastBytes,
const GCSchedulingTunables& tunables,
const GCSchedulingState& state)
{
if (!tunables.isDynamicHeapGrowthEnabled())
return 3.0;
// For small zones, our collection heuristics do not matter much: favor
// something simple in this case.
if (lastBytes < 1 * 1024 * 1024)
return tunables.lowFrequencyHeapGrowth();
// If GC's are not triggering in rapid succession, use a lower threshold so
// that we will collect garbage sooner.
if (!state.inHighFrequencyGCMode())
return tunables.lowFrequencyHeapGrowth();
// The heap growth factor depends on the heap size after a GC and the GC
// frequency. For low frequency GCs (more than 1sec between GCs) we let
// the heap grow to 150%. For high frequency GCs we let the heap grow
// depending on the heap size:
// lastBytes < highFrequencyLowLimit: 300%
// lastBytes > highFrequencyHighLimit: 150%
// otherwise: linear interpolation between 300% and 150% based on lastBytes
// Use shorter names to make the operation comprehensible.
double minRatio = tunables.highFrequencyHeapGrowthMin();
double maxRatio = tunables.highFrequencyHeapGrowthMax();
double lowLimit = tunables.highFrequencyLowLimitBytes();
double highLimit = tunables.highFrequencyHighLimitBytes();
if (lastBytes <= lowLimit)
return maxRatio;
if (lastBytes >= highLimit)
return minRatio;
double factor = maxRatio - ((maxRatio - minRatio) * ((lastBytes - lowLimit) /
(highLimit - lowLimit)));
MOZ_ASSERT(factor >= minRatio);
MOZ_ASSERT(factor <= maxRatio);
return factor;
}
/* static */ size_t
ZoneHeapThreshold::computeZoneTriggerBytes(double growthFactor, size_t lastBytes,
JSGCInvocationKind gckind,
const GCSchedulingTunables& tunables,
const AutoLockGC& lock)
{
size_t base = gckind == GC_SHRINK
? Max(lastBytes, tunables.minEmptyChunkCount(lock) * ChunkSize)
: Max(lastBytes, tunables.gcZoneAllocThresholdBase());
double trigger = double(base) * growthFactor;
return size_t(Min(double(tunables.gcMaxBytes()), trigger));
}
void
ZoneHeapThreshold::updateAfterGC(size_t lastBytes, JSGCInvocationKind gckind,
const GCSchedulingTunables& tunables,
const GCSchedulingState& state, const AutoLockGC& lock)
{
gcHeapGrowthFactor_ = computeZoneHeapGrowthFactorForHeapSize(lastBytes, tunables, state);
gcTriggerBytes_ = computeZoneTriggerBytes(gcHeapGrowthFactor_, lastBytes, gckind, tunables,
lock);
}
void
ZoneHeapThreshold::updateForRemovedArena(const GCSchedulingTunables& tunables)
{
size_t amount = ArenaSize * gcHeapGrowthFactor_;
MOZ_ASSERT(amount > 0);
MOZ_ASSERT(gcTriggerBytes_ >= amount);
if (gcTriggerBytes_ - amount < tunables.gcZoneAllocThresholdBase() * gcHeapGrowthFactor_)
return;
gcTriggerBytes_ -= amount;
}
void
GCMarker::delayMarkingArena(ArenaHeader* aheader)
{
if (aheader->hasDelayedMarking) {
/* Arena already scheduled to be marked later */
return;
}
aheader->setNextDelayedMarking(unmarkedArenaStackTop);
unmarkedArenaStackTop = aheader;
markLaterArenas++;
}
void
GCMarker::delayMarkingChildren(const void* thing)
{
const TenuredCell* cell = TenuredCell::fromPointer(thing);
cell->arenaHeader()->markOverflow = 1;
delayMarkingArena(cell->arenaHeader());
}
inline void
ArenaLists::prepareForIncrementalGC(JSRuntime* rt)
{
for (auto i : AllAllocKinds()) {
FreeList* freeList = &freeLists[i];
if (!freeList->isEmpty()) {
ArenaHeader* aheader = freeList->arenaHeader();
aheader->allocatedDuringIncremental = true;
rt->gc.marker.delayMarkingArena(aheader);
}
}
}
/* Compacting GC */
bool
GCRuntime::shouldCompact()
{
// Compact on shrinking GC if enabled, but skip compacting in incremental
// GCs if we are currently animating.
return invocationKind == GC_SHRINK && isCompactingGCEnabled() &&
(!isIncremental || rt->lastAnimationTime + PRMJ_USEC_PER_SEC < PRMJ_Now());
}
void
GCRuntime::disableCompactingGC()
{
MOZ_ASSERT(CurrentThreadCanAccessRuntime(rt));
++compactingDisabledCount;
}
void
GCRuntime::enableCompactingGC()
{
MOZ_ASSERT(CurrentThreadCanAccessRuntime(rt));
MOZ_ASSERT(compactingDisabledCount > 0);
--compactingDisabledCount;
}
bool
GCRuntime::isCompactingGCEnabled() const
{
MOZ_ASSERT(CurrentThreadCanAccessRuntime(rt));
return compactingEnabled && compactingDisabledCount == 0;
}
AutoDisableCompactingGC::AutoDisableCompactingGC(JSRuntime* rt)
: gc(rt->gc)
{
gc.disableCompactingGC();
}
AutoDisableCompactingGC::~AutoDisableCompactingGC()
{
gc.enableCompactingGC();
}
static bool
CanRelocateZone(Zone* zone)
{
return !zone->isAtomsZone() && !zone->isSelfHostingZone();
}
static bool
CanRelocateAllocKind(AllocKind kind)
{
return IsObjectAllocKind(kind);
}
size_t ArenaHeader::countFreeCells()
{
size_t count = 0;
size_t thingSize = getThingSize();
FreeSpan firstSpan(getFirstFreeSpan());
for (const FreeSpan* span = &firstSpan; !span->isEmpty(); span = span->nextSpan())
count += span->length(thingSize);
return count;
}
size_t ArenaHeader::countUsedCells()
{
return Arena::thingsPerArena(getThingSize()) - countFreeCells();
}
ArenaHeader*
ArenaList::removeRemainingArenas(ArenaHeader** arenap)
{
// This is only ever called to remove arenas that are after the cursor, so
// we don't need to update it.
#ifdef DEBUG
for (ArenaHeader* arena = *arenap; arena; arena = arena->next)
MOZ_ASSERT(cursorp_ != &arena->next);
#endif
ArenaHeader* remainingArenas = *arenap;
*arenap = nullptr;
check();
return remainingArenas;
}
static bool
ShouldRelocateAllArenas(JS::gcreason::Reason reason)
{
return reason == JS::gcreason::DEBUG_GC;
}
/*
* Choose which arenas to relocate all cells from. Return an arena cursor that
* can be passed to removeRemainingArenas().
*/
ArenaHeader**
ArenaList::pickArenasToRelocate(size_t& arenaTotalOut, size_t& relocTotalOut)
{
// Relocate the greatest number of arenas such that the number of used cells
// in relocated arenas is less than or equal to the number of free cells in
// unrelocated arenas. In other words we only relocate cells we can move
// into existing arenas, and we choose the least full areans to relocate.
//
// This is made easier by the fact that the arena list has been sorted in
// descending order of number of used cells, so we will always relocate a
// tail of the arena list. All we need to do is find the point at which to
// start relocating.
check();
if (isCursorAtEnd())
return nullptr;
ArenaHeader** arenap = cursorp_; // Next arena to consider for relocation.
size_t previousFreeCells = 0; // Count of free cells before arenap.
size_t followingUsedCells = 0; // Count of used cells after arenap.
size_t fullArenaCount = 0; // Number of full arenas (not relocated).
size_t nonFullArenaCount = 0; // Number of non-full arenas (considered for relocation).
size_t arenaIndex = 0; // Index of the next arena to consider.
for (ArenaHeader* arena = head_; arena != *cursorp_; arena = arena->next)
fullArenaCount++;
for (ArenaHeader* arena = *cursorp_; arena; arena = arena->next) {
followingUsedCells += arena->countUsedCells();
nonFullArenaCount++;
}
mozilla::DebugOnly<size_t> lastFreeCells(0);
size_t cellsPerArena = Arena::thingsPerArena((*arenap)->getThingSize());
while (*arenap) {
ArenaHeader* arena = *arenap;
if (followingUsedCells <= previousFreeCells)
break;
size_t freeCells = arena->countFreeCells();
size_t usedCells = cellsPerArena - freeCells;
followingUsedCells -= usedCells;
#ifdef DEBUG
MOZ_ASSERT(freeCells >= lastFreeCells);
lastFreeCells = freeCells;
#endif
previousFreeCells += freeCells;
arenap = &arena->next;
arenaIndex++;
}
size_t relocCount = nonFullArenaCount - arenaIndex;
MOZ_ASSERT(relocCount < nonFullArenaCount);
MOZ_ASSERT((relocCount == 0) == (!*arenap));
arenaTotalOut += fullArenaCount + nonFullArenaCount;
relocTotalOut += relocCount;
return arenap;
}
#ifdef DEBUG
inline bool
PtrIsInRange(const void* ptr, const void* start, size_t length)
{
return uintptr_t(ptr) - uintptr_t(start) < length;
}
#endif
static TenuredCell*
AllocRelocatedCell(Zone* zone, AllocKind thingKind, size_t thingSize)
{
AutoEnterOOMUnsafeRegion oomUnsafe;
void* dstAlloc = zone->arenas.allocateFromFreeList(thingKind, thingSize);
if (!dstAlloc)
dstAlloc = GCRuntime::refillFreeListInGC(zone, thingKind);
if (!dstAlloc) {
// This can only happen in zeal mode or debug builds as we don't
// otherwise relocate more cells than we have existing free space
// for.
oomUnsafe.crash("Could not allocate new arena while compacting");
}
return TenuredCell::fromPointer(dstAlloc);
}
static void
RelocateCell(Zone* zone, TenuredCell* src, AllocKind thingKind, size_t thingSize)
{
JS::AutoSuppressGCAnalysis nogc(zone->runtimeFromMainThread());
// Allocate a new cell.
MOZ_ASSERT(zone == src->zone());
TenuredCell* dst = AllocRelocatedCell(zone, thingKind, thingSize);
// Copy source cell contents to destination.
memcpy(dst, src, thingSize);
// Move any uid attached to the object.
src->zone()->transferUniqueId(dst, src);
if (IsObjectAllocKind(thingKind)) {
JSObject* srcObj = static_cast<JSObject*>(static_cast<Cell*>(src));
JSObject* dstObj = static_cast<JSObject*>(static_cast<Cell*>(dst));
if (srcObj->isNative()) {
NativeObject* srcNative = &srcObj->as<NativeObject>();
NativeObject* dstNative = &dstObj->as<NativeObject>();
// Fixup the pointer to inline object elements if necessary.
if (srcNative->hasFixedElements())
dstNative->setFixedElements();
// For copy-on-write objects that own their elements, fix up the
// owner pointer to point to the relocated object.
if (srcNative->denseElementsAreCopyOnWrite()) {
HeapPtrNativeObject& owner = dstNative->getElementsHeader()->ownerObject();
if (owner == srcNative)
owner = dstNative;
}
}
// Call object moved hook if present.
if (JSObjectMovedOp op = srcObj->getClass()->ext.objectMovedOp)
op(dstObj, srcObj);
MOZ_ASSERT_IF(dstObj->isNative(),
!PtrIsInRange((const Value*)dstObj->as<NativeObject>().getDenseElements(),
src, thingSize));
}
// Copy the mark bits.
dst->copyMarkBitsFrom(src);
// Mark source cell as forwarded and leave a pointer to the destination.
RelocationOverlay* overlay = RelocationOverlay::fromCell(src);
overlay->forwardTo(dst);
}
static void
RelocateArena(ArenaHeader* aheader, SliceBudget& sliceBudget)
{
MOZ_ASSERT(aheader->allocated());
MOZ_ASSERT(!aheader->hasDelayedMarking);
MOZ_ASSERT(!aheader->markOverflow);
MOZ_ASSERT(!aheader->allocatedDuringIncremental);
Zone* zone = aheader->zone;
AllocKind thingKind = aheader->getAllocKind();
size_t thingSize = aheader->getThingSize();
for (ArenaCellIterUnderFinalize i(aheader); !i.done(); i.next()) {
RelocateCell(zone, i.getCell(), thingKind, thingSize);
sliceBudget.step();
}
#ifdef DEBUG
for (ArenaCellIterUnderFinalize i(aheader); !i.done(); i.next()) {
TenuredCell* src = i.getCell();
MOZ_ASSERT(RelocationOverlay::isCellForwarded(src));
TenuredCell* dest = Forwarded(src);
MOZ_ASSERT(src->isMarked(BLACK) == dest->isMarked(BLACK));
MOZ_ASSERT(src->isMarked(GRAY) == dest->isMarked(GRAY));
}
#endif
}
static inline bool
ShouldProtectRelocatedArenas(JS::gcreason::Reason reason)
{
// For zeal mode collections we don't release the relocated arenas
// immediately. Instead we protect them and keep them around until the next
// collection so we can catch any stray accesses to them.
#ifdef DEBUG
return reason == JS::gcreason::DEBUG_GC;
#else
return false;
#endif
}
/*
* Relocate all arenas identified by pickArenasToRelocate: for each arena,
* relocate each cell within it, then add it to a list of relocated arenas.
*/
ArenaHeader*
ArenaList::relocateArenas(ArenaHeader* toRelocate, ArenaHeader* relocated, SliceBudget& sliceBudget,
gcstats::Statistics& stats)
{
check();
while (ArenaHeader* arena = toRelocate) {
toRelocate = arena->next;
RelocateArena(arena, sliceBudget);
// Prepend to list of relocated arenas
arena->next = relocated;
relocated = arena;
stats.count(gcstats::STAT_ARENA_RELOCATED);
}
check();
return relocated;
}
// Skip compacting zones unless we can free a certain proportion of their GC
// heap memory.
static const double MIN_ZONE_RECLAIM_PERCENT = 2.0;
static bool
IsOOMReason(JS::gcreason::Reason reason)
{
return reason == JS::gcreason::LAST_DITCH ||
reason == JS::gcreason::MEM_PRESSURE;
}
static bool
ShouldRelocateZone(size_t arenaCount, size_t relocCount, JS::gcreason::Reason reason)
{
if (relocCount == 0)
return false;
if (IsOOMReason(reason))
return true;
return (relocCount * 100.0) / arenaCount >= MIN_ZONE_RECLAIM_PERCENT;
}
bool
ArenaLists::relocateArenas(Zone* zone, ArenaHeader*& relocatedListOut, JS::gcreason::Reason reason,
SliceBudget& sliceBudget, gcstats::Statistics& stats)
{
// This is only called from the main thread while we are doing a GC, so
// there is no need to lock.
MOZ_ASSERT(CurrentThreadCanAccessRuntime(runtime_));
MOZ_ASSERT(runtime_->gc.isHeapCompacting());
MOZ_ASSERT(!runtime_->gc.isBackgroundSweeping());
// Flush all the freeLists back into the arena headers
purge();
checkEmptyFreeLists();
if (ShouldRelocateAllArenas(reason)) {
zone->prepareForCompacting();
for (auto i : AllAllocKinds()) {
if (CanRelocateAllocKind(i)) {
ArenaList& al = arenaLists[i];
ArenaHeader* allArenas = al.head();
al.clear();
relocatedListOut = al.relocateArenas(allArenas, relocatedListOut, sliceBudget, stats);
}
}
} else {
size_t arenaCount = 0;
size_t relocCount = 0;
AllAllocKindArray<ArenaHeader**> toRelocate;
for (auto i : AllAllocKinds()) {
toRelocate[i] = nullptr;
if (CanRelocateAllocKind(i))
toRelocate[i] = arenaLists[i].pickArenasToRelocate(arenaCount, relocCount);
}
if (!ShouldRelocateZone(arenaCount, relocCount, reason))
return false;
zone->prepareForCompacting();
for (auto i : AllAllocKinds()) {
if (toRelocate[i]) {
ArenaList& al = arenaLists[i];
ArenaHeader* arenas = al.removeRemainingArenas(toRelocate[i]);
relocatedListOut = al.relocateArenas(arenas, relocatedListOut, sliceBudget, stats);
}
}
}
// When we allocate new locations for cells, we use
// allocateFromFreeList(). Reset the free list again so that
// AutoCopyFreeListToArenasForGC doesn't complain that the free lists are
// different now.
purge();
checkEmptyFreeLists();
return true;
}
bool
GCRuntime::relocateArenas(Zone* zone, JS::gcreason::Reason reason, ArenaHeader*& relocatedListOut,
SliceBudget& sliceBudget)
{
gcstats::AutoPhase ap(stats, gcstats::PHASE_COMPACT_MOVE);
MOZ_ASSERT(!zone->isPreservingCode());
MOZ_ASSERT(CanRelocateZone(zone));
jit::StopAllOffThreadCompilations(zone);
if (!zone->arenas.relocateArenas(zone, relocatedListOut, reason, sliceBudget, stats))
return false;
#ifdef DEBUG
// Check that we did as much compaction as we should have. There
// should always be less than one arena's worth of free cells.
for (auto i : AllAllocKinds()) {
size_t thingsPerArena = Arena::thingsPerArena(Arena::thingSize(i));
if (CanRelocateAllocKind(i)) {
ArenaList& al = zone->arenas.arenaLists[i];
size_t freeCells = 0;
for (ArenaHeader* arena = al.arenaAfterCursor(); arena; arena = arena->next)
freeCells += arena->countFreeCells();
MOZ_ASSERT(freeCells < thingsPerArena);
}
}
#endif
return true;
}
void
MovingTracer::onObjectEdge(JSObject** objp)
{
if (IsForwarded(*objp))
*objp = Forwarded(*objp);
}
void
Zone::prepareForCompacting()
{
FreeOp* fop = runtimeFromMainThread()->defaultFreeOp();
discardJitCode(fop);
}
void
GCRuntime::sweepTypesAfterCompacting(Zone* zone)
{
FreeOp* fop = rt->defaultFreeOp();
zone->beginSweepTypes(fop, rt->gc.releaseObservedTypes && !zone->isPreservingCode());
AutoClearTypeInferenceStateOnOOM oom(zone);
for (ZoneCellIterUnderGC i(zone, AllocKind::SCRIPT); !i.done(); i.next()) {
JSScript* script = i.get<JSScript>();
script->maybeSweepTypes(&oom);
}
for (ZoneCellIterUnderGC i(zone, AllocKind::OBJECT_GROUP); !i.done(); i.next()) {
ObjectGroup* group = i.get<ObjectGroup>();
group->maybeSweep(&oom);
}
zone->types.endSweep(rt);
}
void
GCRuntime::sweepZoneAfterCompacting(Zone* zone)
{
MOZ_ASSERT(zone->isCollecting());
FreeOp* fop = rt->defaultFreeOp();
sweepTypesAfterCompacting(zone);
zone->sweepBreakpoints(fop);
zone->sweepWeakMaps();
for (CompartmentsInZoneIter c(zone); !c.done(); c.next()) {
c->sweepInnerViews();
c->sweepBaseShapeTable();
c->sweepInitialShapeTable();
c->objectGroups.sweep(fop);
c->sweepRegExps();
c->sweepSavedStacks();
c->sweepGlobalObject(fop);
c->sweepObjectPendingMetadata();
c->sweepSelfHostingScriptSource();
c->sweepDebugScopes();
c->sweepJitCompartment(fop);
c->sweepNativeIterators();
c->sweepTemplateObjects();
}
}
template <typename T>
static void
UpdateCellPointersTyped(MovingTracer* trc, ArenaHeader* arena, JS::TraceKind traceKind)
{
for (ArenaCellIterUnderGC i(arena); !i.done(); i.next()) {
T* cell = reinterpret_cast<T*>(i.getCell());
cell->fixupAfterMovingGC();
TraceChildren(trc, cell, traceKind);
}
}
/*
* Update the interal pointers for all cells in an arena.
*/
static void
UpdateCellPointers(MovingTracer* trc, ArenaHeader* arena)
{
AllocKind kind = arena->getAllocKind();
JS::TraceKind traceKind = MapAllocToTraceKind(kind);
switch (kind) {
case AllocKind::FUNCTION:
case AllocKind::FUNCTION_EXTENDED:
case AllocKind::OBJECT0:
case AllocKind::OBJECT0_BACKGROUND:
case AllocKind::OBJECT2:
case AllocKind::OBJECT2_BACKGROUND:
case AllocKind::OBJECT4:
case AllocKind::OBJECT4_BACKGROUND:
case AllocKind::OBJECT8:
case AllocKind::OBJECT8_BACKGROUND:
case AllocKind::OBJECT12:
case AllocKind::OBJECT12_BACKGROUND:
case AllocKind::OBJECT16:
case AllocKind::OBJECT16_BACKGROUND:
UpdateCellPointersTyped<JSObject>(trc, arena, traceKind);
return;
case AllocKind::SCRIPT:
UpdateCellPointersTyped<JSScript>(trc, arena, traceKind);
return;
case AllocKind::LAZY_SCRIPT:
UpdateCellPointersTyped<LazyScript>(trc, arena, traceKind);
return;
case AllocKind::SHAPE:
UpdateCellPointersTyped<Shape>(trc, arena, traceKind);
return;
case AllocKind::ACCESSOR_SHAPE:
UpdateCellPointersTyped<AccessorShape>(trc, arena, traceKind);
return;
case AllocKind::BASE_SHAPE:
UpdateCellPointersTyped<BaseShape>(trc, arena, traceKind);
return;
case AllocKind::OBJECT_GROUP:
UpdateCellPointersTyped<ObjectGroup>(trc, arena, traceKind);
return;
case AllocKind::JITCODE:
UpdateCellPointersTyped<jit::JitCode>(trc, arena, traceKind);
return;
default:
MOZ_CRASH("Invalid alloc kind for UpdateCellPointers");
}
}
namespace js {
namespace gc {
struct ArenasToUpdate
{
enum KindsToUpdate {
FOREGROUND = 1,
BACKGROUND = 2,
ALL = FOREGROUND | BACKGROUND
};
ArenasToUpdate(Zone* zone, KindsToUpdate kinds);
bool done() { return kind == AllocKind::LIMIT; }
ArenaHeader* getArenasToUpdate(AutoLockHelperThreadState& lock, unsigned max);
private:
KindsToUpdate kinds; // Selects which thing kinds to iterate
Zone* zone; // Zone to process
AllocKind kind; // Current alloc kind to process
ArenaHeader* arena; // Next arena to process
AllocKind nextAllocKind(AllocKind i) { return AllocKind(uint8_t(i) + 1); }
bool shouldProcessKind(AllocKind kind);
ArenaHeader* next(AutoLockHelperThreadState& lock);
};
bool ArenasToUpdate::shouldProcessKind(AllocKind kind)
{
MOZ_ASSERT(IsValidAllocKind(kind));
// GC things that do not contain JSObject pointers don't need updating.
if (kind == AllocKind::FAT_INLINE_STRING ||
kind == AllocKind::STRING ||
kind == AllocKind::EXTERNAL_STRING ||
kind == AllocKind::SYMBOL)
{
return false;
}
// We try to update as many GC things in parallel as we can, but there are
// kinds for which this might not be safe:
// - we assume JSObjects that are foreground finalized are not safe to
// update in parallel
// - updating a shape touches child shapes in fixupShapeTreeAfterMovingGC()
if (js::gc::IsBackgroundFinalized(kind) &&
kind != AllocKind::SHAPE &&
kind != AllocKind::ACCESSOR_SHAPE)
{
return (kinds & BACKGROUND) != 0;
} else {
return (kinds & FOREGROUND) != 0;
}
}
ArenasToUpdate::ArenasToUpdate(Zone* zone, KindsToUpdate kinds)
: kinds(kinds), zone(zone), kind(AllocKind::FIRST), arena(nullptr)
{
MOZ_ASSERT(zone->isGCCompacting());
MOZ_ASSERT(kinds && !(kinds & ~ALL));
}
ArenaHeader*
ArenasToUpdate::next(AutoLockHelperThreadState& lock)
{
// Find the next arena to update.
//
// This iterates through the GC thing kinds filtered by shouldProcessKind(),
// and then through thea arenas of that kind. All state is held in the
// object and we just return when we find an arena.
for (; kind < AllocKind::LIMIT; kind = nextAllocKind(kind)) {
if (shouldProcessKind(kind)) {
if (!arena)
arena = zone->arenas.getFirstArena(kind);
else
arena = arena->next;
if (arena)
return arena;
}
}
MOZ_ASSERT(!arena);
MOZ_ASSERT(done());
return nullptr;
}
ArenaHeader*
ArenasToUpdate::getArenasToUpdate(AutoLockHelperThreadState& lock, unsigned count)
{
if (done())
return nullptr;
ArenaHeader* head = nullptr;
ArenaHeader* tail = nullptr;
for (unsigned i = 0; i < count; ++i) {
ArenaHeader* arena = next(lock);
if (!arena)
break;
if (tail)
tail->setNextArenaToUpdate(arena);
else
head = arena;
tail = arena;
}
return head;
}
struct UpdateCellPointersTask : public GCParallelTask
{
// Number of arenas to update in one block.
#ifdef DEBUG
static const unsigned ArenasToProcess = 16;
#else
static const unsigned ArenasToProcess = 256;
#endif
UpdateCellPointersTask() : rt_(nullptr), source_(nullptr), arenaList_(nullptr) {}
void init(JSRuntime* rt, ArenasToUpdate* source, AutoLockHelperThreadState& lock);
~UpdateCellPointersTask() override { join(); }
private:
JSRuntime* rt_;
ArenasToUpdate* source_;
ArenaHeader* arenaList_;
virtual void run() override;
void getArenasToUpdate(AutoLockHelperThreadState& lock);
void updateArenas();
};
void
UpdateCellPointersTask::init(JSRuntime* rt, ArenasToUpdate* source, AutoLockHelperThreadState& lock)
{
rt_ = rt;
source_ = source;
getArenasToUpdate(lock);
}
void
UpdateCellPointersTask::getArenasToUpdate(AutoLockHelperThreadState& lock)
{
arenaList_ = source_->getArenasToUpdate(lock, ArenasToProcess);
}
void
UpdateCellPointersTask::updateArenas()
{
MovingTracer trc(rt_);
for (ArenaHeader* arena = arenaList_;
arena;
arena = arena->getNextArenaToUpdateAndUnlink())
{
UpdateCellPointers(&trc, arena);
}
arenaList_ = nullptr;
}
/* virtual */ void
UpdateCellPointersTask::run()
{
MOZ_ASSERT(!HelperThreadState().isLocked());
while (arenaList_) {
updateArenas();
{
AutoLockHelperThreadState lock;
getArenasToUpdate(lock);
}
}
}
} // namespace gc
} // namespace js
void
GCRuntime::updateAllCellPointersParallel(MovingTracer* trc, Zone* zone)
{
AutoDisableProxyCheck noProxyCheck(rt); // These checks assert when run in parallel.
const size_t minTasks = 2;
const size_t maxTasks = 8;
size_t targetTaskCount = HelperThreadState().cpuCount / 2;
size_t taskCount = Min(Max(targetTaskCount, minTasks), maxTasks);
UpdateCellPointersTask bgTasks[maxTasks];
UpdateCellPointersTask fgTask;
ArenasToUpdate fgArenas(zone, ArenasToUpdate::FOREGROUND);
ArenasToUpdate bgArenas(zone, ArenasToUpdate::BACKGROUND);
unsigned tasksStarted = 0;
{
AutoLockHelperThreadState lock;
unsigned i;
for (i = 0; i < taskCount && !bgArenas.done(); ++i) {
bgTasks[i].init(rt, &bgArenas, lock);
startTask(bgTasks[i], gcstats::PHASE_COMPACT_UPDATE_CELLS);
}
tasksStarted = i;
fgTask.init(rt, &fgArenas, lock);
}
fgTask.runFromMainThread(rt);
{
AutoLockHelperThreadState lock;
for (unsigned i = 0; i < tasksStarted; ++i)
joinTask(bgTasks[i], gcstats::PHASE_COMPACT_UPDATE_CELLS);
}
}
void
GCRuntime::updateAllCellPointersSerial(MovingTracer* trc, Zone* zone)
{
UpdateCellPointersTask task;
{
AutoLockHelperThreadState lock;
ArenasToUpdate allArenas(zone, ArenasToUpdate::ALL);
task.init(rt, &allArenas, lock);
}
task.runFromMainThread(rt);
}
/*
* Update pointers to relocated cells by doing a full heap traversal and sweep.
*
* The latter is necessary to update weak references which are not marked as
* part of the traversal.
*/
void
GCRuntime::updatePointersToRelocatedCells(Zone* zone)
{
MOZ_ASSERT(zone->isGCCompacting());
MOZ_ASSERT(rt->currentThreadHasExclusiveAccess());
gcstats::AutoPhase ap(stats, gcstats::PHASE_COMPACT_UPDATE);
MovingTracer trc(rt);
// Fixup compartment global pointers as these get accessed during marking.
for (CompartmentsInZoneIter comp(zone); !comp.done(); comp.next())
comp->fixupAfterMovingGC();
JSCompartment::fixupCrossCompartmentWrappersAfterMovingGC(&trc);
// Mark roots to update them.
{
markRuntime(&trc, MarkRuntime);
gcstats::AutoPhase ap(stats, gcstats::PHASE_MARK_ROOTS);
Debugger::markAll(&trc);
Debugger::markIncomingCrossCompartmentEdges(&trc);
WeakMapBase::markAll(zone, &trc);
for (CompartmentsInZoneIter c(zone); !c.done(); c.next()) {
c->trace(&trc);
if (c->watchpointMap)
c->watchpointMap->markAll(&trc);
}
// Mark all gray roots, making sure we call the trace callback to get the
// current set.
if (JSTraceDataOp op = grayRootTracer.op)
(*op)(&trc, grayRootTracer.data);
}
// Sweep everything to fix up weak pointers
WatchpointMap::sweepAll(rt);
Debugger::sweepAll(rt->defaultFreeOp());
jit::JitRuntime::SweepJitcodeGlobalTable(rt);
rt->gc.sweepZoneAfterCompacting(zone);
// Type inference may put more blocks here to free.
freeLifoAlloc.freeAll();
// Clear runtime caches that can contain cell pointers.
// TODO: Should possibly just call purgeRuntime() here.
rt->newObjectCache.purge();
rt->nativeIterCache.purge();
// Call callbacks to get the rest of the system to fixup other untraced pointers.
callWeakPointerZoneGroupCallbacks();
for (CompartmentsInZoneIter comp(zone); !comp.done(); comp.next())
callWeakPointerCompartmentCallbacks(comp);
// Finally, iterate through all cells that can contain JSObject pointers to
// update them. Since updating each cell is independent we try to
// parallelize this as much as possible.
if (CanUseExtraThreads())
updateAllCellPointersParallel(&trc, zone);
else
updateAllCellPointersSerial(&trc, zone);
}
void
GCRuntime::protectAndHoldArenas(ArenaHeader* arenaList)
{
for (ArenaHeader* arena = arenaList; arena; ) {
MOZ_ASSERT(arena->allocated());
ArenaHeader* next = arena->next;
if (!next) {
// Prepend to hold list before we protect the memory.
arena->next = relocatedArenasToRelease;
relocatedArenasToRelease = arenaList;
}
ProtectPages(arena, ArenaSize);
arena = next;
}
}
void
GCRuntime::unprotectHeldRelocatedArenas()
{
for (ArenaHeader* arena = relocatedArenasToRelease; arena; arena = arena->next) {
UnprotectPages(arena, ArenaSize);
MOZ_ASSERT(arena->allocated());
}
}
void
GCRuntime::releaseRelocatedArenas(ArenaHeader* arenaList)
{
AutoLockGC lock(rt);
releaseRelocatedArenasWithoutUnlocking(arenaList, lock);
expireChunksAndArenas(true, lock);
}
void
GCRuntime::releaseRelocatedArenasWithoutUnlocking(ArenaHeader* arenaList, const AutoLockGC& lock)
{
// Release the relocated arenas, now containing only forwarding pointers
unsigned count = 0;
while (arenaList) {
ArenaHeader* aheader = arenaList;
arenaList = arenaList->next;
// Clear the mark bits
aheader->unmarkAll();
// Mark arena as empty
AllocKind thingKind = aheader->getAllocKind();
size_t thingSize = aheader->getThingSize();
Arena* arena = aheader->getArena();
FreeSpan fullSpan;
fullSpan.initFinal(arena->thingsStart(thingKind), arena->thingsEnd() - thingSize, thingSize);
aheader->setFirstFreeSpan(&fullSpan);
#if defined(JS_CRASH_DIAGNOSTICS) || defined(JS_GC_ZEAL)
JS_POISON(reinterpret_cast<void*>(arena->thingsStart(thingKind)),
JS_MOVED_TENURED_PATTERN, Arena::thingsSpan(thingSize));
#endif
releaseArena(aheader, lock);
++count;
}
}
// In debug mode we don't always release relocated arenas straight away.
// Sometimes protect them instead and hold onto them until the next GC sweep
// phase to catch any pointers to them that didn't get forwarded.
void
GCRuntime::releaseHeldRelocatedArenas()
{
#ifdef DEBUG
unprotectHeldRelocatedArenas();
releaseRelocatedArenas(relocatedArenasToRelease);
relocatedArenasToRelease = nullptr;
#endif
}
void
GCRuntime::releaseHeldRelocatedArenasWithoutUnlocking(const AutoLockGC& lock)
{
#ifdef DEBUG
unprotectHeldRelocatedArenas();
releaseRelocatedArenasWithoutUnlocking(relocatedArenasToRelease, lock);
relocatedArenasToRelease = nullptr;
#endif
}
void
ReleaseArenaList(JSRuntime* rt, ArenaHeader* aheader, const AutoLockGC& lock)
{
ArenaHeader* next;
for (; aheader; aheader = next) {
next = aheader->next;
rt->gc.releaseArena(aheader, lock);
}
}
ArenaLists::~ArenaLists()
{
AutoLockGC lock(runtime_);
for (auto i : AllAllocKinds()) {
/*
* We can only call this during the shutdown after the last GC when
* the background finalization is disabled.
*/
MOZ_ASSERT(backgroundFinalizeState[i] == BFS_DONE);
ReleaseArenaList(runtime_, arenaLists[i].head(), lock);
}
ReleaseArenaList(runtime_, incrementalSweptArenas.head(), lock);
for (auto i : ObjectAllocKinds())
ReleaseArenaList(runtime_, savedObjectArenas[i].head(), lock);
ReleaseArenaList(runtime_, savedEmptyObjectArenas, lock);
}
void
ArenaLists::finalizeNow(FreeOp* fop, const FinalizePhase& phase)
{
gcstats::AutoPhase ap(fop->runtime()->gc.stats, phase.statsPhase);
for (unsigned i = 0; i < phase.length; ++i)
finalizeNow(fop, phase.kinds[i], RELEASE_ARENAS, nullptr);
}
void
ArenaLists::finalizeNow(FreeOp* fop, AllocKind thingKind, KeepArenasEnum keepArenas, ArenaHeader** empty)
{
MOZ_ASSERT(!IsBackgroundFinalized(thingKind));
forceFinalizeNow(fop, thingKind, keepArenas, empty);
}
void
ArenaLists::forceFinalizeNow(FreeOp* fop, AllocKind thingKind, KeepArenasEnum keepArenas, ArenaHeader** empty)
{
MOZ_ASSERT(backgroundFinalizeState[thingKind] == BFS_DONE);
ArenaHeader* arenas = arenaLists[thingKind].head();
if (!arenas)
return;
arenaLists[thingKind].clear();
size_t thingsPerArena = Arena::thingsPerArena(Arena::thingSize(thingKind));
SortedArenaList finalizedSorted(thingsPerArena);
auto unlimited = SliceBudget::unlimited();
FinalizeArenas(fop, &arenas, finalizedSorted, thingKind, unlimited, keepArenas);
MOZ_ASSERT(!arenas);
if (empty) {
MOZ_ASSERT(keepArenas == KEEP_ARENAS);
finalizedSorted.extractEmpty(empty);
}
arenaLists[thingKind] = finalizedSorted.toArenaList();
}
void
ArenaLists::queueForForegroundSweep(FreeOp* fop, const FinalizePhase& phase)
{
gcstats::AutoPhase ap(fop->runtime()->gc.stats, phase.statsPhase);
for (unsigned i = 0; i < phase.length; ++i)
queueForForegroundSweep(fop, phase.kinds[i]);
}
void
ArenaLists::queueForForegroundSweep(FreeOp* fop, AllocKind thingKind)
{
MOZ_ASSERT(!IsBackgroundFinalized(thingKind));
MOZ_ASSERT(backgroundFinalizeState[thingKind] == BFS_DONE);
MOZ_ASSERT(!arenaListsToSweep[thingKind]);
arenaListsToSweep[thingKind] = arenaLists[thingKind].head();
arenaLists[thingKind].clear();
}
void
ArenaLists::queueForBackgroundSweep(FreeOp* fop, const FinalizePhase& phase)
{
gcstats::AutoPhase ap(fop->runtime()->gc.stats, phase.statsPhase);
for (unsigned i = 0; i < phase.length; ++i)
queueForBackgroundSweep(fop, phase.kinds[i]);
}
inline void
ArenaLists::queueForBackgroundSweep(FreeOp* fop, AllocKind thingKind)
{
MOZ_ASSERT(IsBackgroundFinalized(thingKind));
ArenaList* al = &arenaLists[thingKind];
if (al->isEmpty()) {
MOZ_ASSERT(backgroundFinalizeState[thingKind] == BFS_DONE);
return;
}
MOZ_ASSERT(backgroundFinalizeState[thingKind] == BFS_DONE);
arenaListsToSweep[thingKind] = al->head();
al->clear();
backgroundFinalizeState[thingKind] = BFS_RUN;
}
/*static*/ void
ArenaLists::backgroundFinalize(FreeOp* fop, ArenaHeader* listHead, ArenaHeader** empty)
{
MOZ_ASSERT(listHead);
MOZ_ASSERT(empty);
AllocKind thingKind = listHead->getAllocKind();
Zone* zone = listHead->zone;
size_t thingsPerArena = Arena::thingsPerArena(Arena::thingSize(thingKind));
SortedArenaList finalizedSorted(thingsPerArena);
auto unlimited = SliceBudget::unlimited();
FinalizeArenas(fop, &listHead, finalizedSorted, thingKind, unlimited, KEEP_ARENAS);
MOZ_ASSERT(!listHead);
finalizedSorted.extractEmpty(empty);
// When arenas are queued for background finalization, all arenas are moved
// to arenaListsToSweep[], leaving the arenaLists[] empty. However, new
// arenas may be allocated before background finalization finishes; now that
// finalization is complete, we want to merge these lists back together.
ArenaLists* lists = &zone->arenas;
ArenaList* al = &lists->arenaLists[thingKind];
// Flatten |finalizedSorted| into a regular ArenaList.
ArenaList finalized = finalizedSorted.toArenaList();
// We must take the GC lock to be able to safely modify the ArenaList;
// however, this does not by itself make the changes visible to all threads,
// as not all threads take the GC lock to read the ArenaLists.
// That safety is provided by the ReleaseAcquire memory ordering of the
// background finalize state, which we explicitly set as the final step.
{
AutoLockGC lock(fop->runtime());
MOZ_ASSERT(lists->backgroundFinalizeState[thingKind] == BFS_RUN);
// Join |al| and |finalized| into a single list.
*al = finalized.insertListWithCursorAtEnd(*al);
lists->arenaListsToSweep[thingKind] = nullptr;
}
lists->backgroundFinalizeState[thingKind] = BFS_DONE;
}
void
ArenaLists::queueForegroundObjectsForSweep(FreeOp* fop)
{
gcstats::AutoPhase ap(fop->runtime()->gc.stats, gcstats::PHASE_SWEEP_OBJECT);
#ifdef DEBUG
for (auto i : ObjectAllocKinds()) { // Braces needed to appease MSVC 2013.
MOZ_ASSERT(savedObjectArenas[i].isEmpty());
}
MOZ_ASSERT(savedEmptyObjectArenas == nullptr);
#endif
// Foreground finalized objects must be finalized at the beginning of the
// sweep phase, before control can return to the mutator. Otherwise,
// mutator behavior can resurrect certain objects whose references would
// otherwise have been erased by the finalizer.
finalizeNow(fop, AllocKind::OBJECT0, KEEP_ARENAS, &savedEmptyObjectArenas);
finalizeNow(fop, AllocKind::OBJECT2, KEEP_ARENAS, &savedEmptyObjectArenas);
finalizeNow(fop, AllocKind::OBJECT4, KEEP_ARENAS, &savedEmptyObjectArenas);
finalizeNow(fop, AllocKind::OBJECT8, KEEP_ARENAS, &savedEmptyObjectArenas);
finalizeNow(fop, AllocKind::OBJECT12, KEEP_ARENAS, &savedEmptyObjectArenas);
finalizeNow(fop, AllocKind::OBJECT16, KEEP_ARENAS, &savedEmptyObjectArenas);
// Prevent the arenas from having new objects allocated into them. We need
// to know which objects are marked while we incrementally sweep dead
// references from type information.
savedObjectArenas[AllocKind::OBJECT0] = arenaLists[AllocKind::OBJECT0].copyAndClear();
savedObjectArenas[AllocKind::OBJECT2] = arenaLists[AllocKind::OBJECT2].copyAndClear();
savedObjectArenas[AllocKind::OBJECT4] = arenaLists[AllocKind::OBJECT4].copyAndClear();
savedObjectArenas[AllocKind::OBJECT8] = arenaLists[AllocKind::OBJECT8].copyAndClear();
savedObjectArenas[AllocKind::OBJECT12] = arenaLists[AllocKind::OBJECT12].copyAndClear();
savedObjectArenas[AllocKind::OBJECT16] = arenaLists[AllocKind::OBJECT16].copyAndClear();
}
void
ArenaLists::mergeForegroundSweptObjectArenas()
{
AutoLockGC lock(runtime_);
ReleaseArenaList(runtime_, savedEmptyObjectArenas, lock);
savedEmptyObjectArenas = nullptr;
mergeSweptArenas(AllocKind::OBJECT0);
mergeSweptArenas(AllocKind::OBJECT2);
mergeSweptArenas(AllocKind::OBJECT4);
mergeSweptArenas(AllocKind::OBJECT8);
mergeSweptArenas(AllocKind::OBJECT12);
mergeSweptArenas(AllocKind::OBJECT16);
}
inline void
ArenaLists::mergeSweptArenas(AllocKind thingKind)
{
ArenaList* al = &arenaLists[thingKind];
ArenaList* saved = &savedObjectArenas[thingKind];
*al = saved->insertListWithCursorAtEnd(*al);
saved->clear();
}
void
ArenaLists::queueForegroundThingsForSweep(FreeOp* fop)
{
gcShapeArenasToUpdate = arenaListsToSweep[AllocKind::SHAPE];
gcAccessorShapeArenasToUpdate = arenaListsToSweep[AllocKind::ACCESSOR_SHAPE];
gcObjectGroupArenasToUpdate = arenaListsToSweep[AllocKind::OBJECT_GROUP];
gcScriptArenasToUpdate = arenaListsToSweep[AllocKind::SCRIPT];
}
/* static */ void*
GCRuntime::refillFreeListInGC(Zone* zone, AllocKind thingKind)
{
/*
* Called by compacting GC to refill a free list while we are in a GC.
*/
MOZ_ASSERT(zone->arenas.freeLists[thingKind].isEmpty());
mozilla::DebugOnly<JSRuntime*> rt = zone->runtimeFromMainThread();
MOZ_ASSERT(rt->isHeapMajorCollecting());
MOZ_ASSERT(!rt->gc.isBackgroundSweeping());
AutoMaybeStartBackgroundAllocation maybeStartBackgroundAllocation;
return zone->arenas.allocateFromArena(zone, thingKind, maybeStartBackgroundAllocation);
}
SliceBudget::SliceBudget()
: timeBudget(UnlimitedTimeBudget), workBudget(UnlimitedWorkBudget)
{
makeUnlimited();