Snakemake crashes with "JSONDecodeError: Extra data:"

Question

[RNG_Daemon]

I am running a snakemake pipeline for variant calling. It executes the jobs just fine, but when the jobs finish, snakemake crashes while trying to schedule new jobs. I am not using any JSON files in my analysis.

Traceback (most recent call last):
  File "/python/lib/python3.9/site-packages/snakemake/__init__.py", line 701, in snakemake
    success = workflow.execute(
  File "/python/lib/python3.9/site-packages/snakemake/workflow.py", line 1075, in execute
    success = self.scheduler.schedule()
  File "/python/lib/python3.9/site-packages/snakemake/scheduler.py", line 440, in schedule
    self._finish_jobs()
  File "/python/lib/python3.9/site-packages/snakemake/scheduler.py", line 529, in _finish_jobs
    self.get_executor(job).handle_job_success(job)
  File "/python/lib/python3.9/site-packages/snakemake/executors/__init__.py", line 876, in handle_job_success
    super().handle_job_success(
  File "/python/lib/python3.9/site-packages/snakemake/executors/__init__.py", line 265, in handle_job_success
    job.postprocess(
  File "/python/lib/python3.9/site-packages/snakemake/jobs.py", line 1032, in postprocess
    self.dag.workflow.persistence.finished(
  File "/python/lib/python3.9/site-packages/snakemake/persistence.py", line 239, in finished
    starttime = self._read_record(self._metadata_path, f).get(
  File "/python/lib/python3.9/site-packages/snakemake/persistence.py", line 414, in _read_record_uncached
    return json.load(f)
  File "/python/lib/python3.9/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/python/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/python/lib/python3.9/json/decoder.py", line 340, in decode
    raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 1 column 1366 (char 1365)
Finished

• snakemake: 6.15.1
• conda: 4.11.0
• slurm: 21.08.4

[Wayne]

Looks similar to this issue report, I think?

[RNG_Daemon]

Yes, this definitely seems related.

It seems I could "fix" it by deleting everything in the .snakemake directory.