Creating a knowledge base for a custom GPT
Authors
Affiliations
NeuroPoly, Polytechnique Montreal, Quebec, Canada
Montreal Heart Institute, Montreal, Quebec, Canada
NeuroPoly, Polytechnique Montreal, Quebec, Canada
NeuroPoly, Polytechnique Montreal, Quebec, Canada
Montreal Heart Institute, Montreal, Quebec, Canada
Center for Advanced Interdisciplinary Research, Ss. Cyril and Methodius University, Skopje, North Macedonia
1Creating a knowledge base for a custom GPT¶
We created a custom GPT (Generative Pretrained Transformer) model, designed specifically to assist in the analysis and synthesis of information pertaining to the 31 reproducible research insights. The knowledge base of this retrieval-augmented generation framework comprises GPT-4 summaries of RRInsights interviews coupled with the original abstract, title, and keywords of the respective articles. This specialized GPT, named RRInsights, is tailored to process and interpret the provided data in the context of reproducibility. Through its advanced natural language processing capabilities, RRInsights can efficiently analyze the scoped literature, extract key insights, and generate comprehensive overviews of the research papers focusing on MRI technology.
import os
from openai import OpenAI
import pickle
from dotenv import dotenv_values
import pandas as pd
import tiktoken
import numpy as np
from sklearn.manifold import TSNE
import plotly.graph_objects as go
from plotly.offline import plot
from IPython.display import display, HTML
import plotly.io as pio
pio.renderers.default = "plotly_mimetype"
config = dotenv_values("../.env")
DATA_ROOT = "../data/repro-mri-scoping/repro_mri_scoping"
try:
client = OpenAI(api_key=config['OPENAI_KEY'])
except:
config = {}
config['OPENAI_KEY'] = False
def read_file(file_path):
with open(file_path, 'r', encoding='utf-8') as file:
content = file.read()
return content
def write_file(file_path, content):
with open(file_path, 'w') as file:
file.write(content)
def write_pickle(filename,content):
with open(filename, 'wb') as file:
pickle.dump(content, file)
def read_pickle(file_name):
with open(file_name, 'rb') as f:
data = pickle.load(f)
return data
def get_output_dir(file_name):
op_dir = "../output"
if not os.path.exists(op_dir):
os.mkdir(op_dir)
return os.path.join(op_dir,file_name)
def get_embedding(text, model="text-embedding-ada-002"):
text = text.replace("\n", " ")
return client.embeddings.create(input = [text], model=model).data[0].embeddingdirectory_path = os.path.join(DATA_ROOT,"repro_insights_parsed_nov23")
# Read the data that has been crawled and parsed.
input_files = [os.path.join(directory_path,f) for f in os.listdir(directory_path) if f.endswith('.txt')]
# Task definition for GPT
messages = [{"role": "system", "content": "You are a helpful assistant designed to understand and summarize the reproducibility aspects of neuroimaging research articles."}]
"""
!!!! WARNING !!!!
The following code requires GPT tokens to
feed insight files one by one and to return
a summary. Estimated cost ~6-8 USD.
"""
if config['OPENAI_KEY']:
# Comment out this line to run it on all the entries.
input_files = input_files[0:2]
for ii, cur_file in enumerate(input_files):
cur_content = read_file(cur_file)
print(f"Requesting summary for {cur_file}")
messages.append({"role": "user", "content": f"The following content is from ARTICLE {ii+1}. In relavance to its title and abstract, create 4 keywords, then summarize what did the study achieve and why it was reproducible in 4 sentences: {cur_content}"})
response = client.chat.completions.create(
model="gpt-4-1106-preview",
messages= messages
)
try:
cur_response = {"role": "assistant", "content": response.choices[0].message.content}
messages.append(cur_response)
write_pickle(get_output_dir("gpt_chat.pkl"),messages)
except Exception as e:
print(f"Exception: {e}")
write_pickle(get_output_dir("gpt_chat.pkl"),messages)
gpt_outputs = os.path.join(get_output_dir("gpt_chat.pkl"))
else:
print('API key not found. Available dataset will be used. Example: \n')
gpt_outputs = os.path.join(DATA_ROOT, "gpt_chat.pkl")
messages = read_pickle(gpt_outputs)
print(messages[0:3])API key not found. Available dataset will be used. Example:
[{'role': 'system', 'content': 'You are a helpful assistant designed to understand and summarize the reproducibility aspects of neuroimaging research articles.'}, {'role': 'user', 'content': 'The following content is from ARTICLE 1. In relavance to its title and abstract, create 4 keywords, then summarize what did the study achieve and why it was reproducible in 4 sentences: Title: In vivo magnetic resonance 31P‐Spectral Analysis With Neural Networks: 31P‐SPAWNN \nAbstract: We have introduced an artificial intelligence framework, 31P‐SPAWNN, in order to fully analyze phosphorus‐31 ( 31$$ {}^{31} $$ P) magnetic resonance spectra. The flexibility and speed of the technique rival traditional least‐square fitting methods, with the performance of the two approaches, are compared in this work. \nReproducibility Insights:\nGeneral questions\nQuestions about the specific reproducible research habit\n\nThis MRM Reproducible Research Insights interview is with Julien Songeon and Antoine Klauser from the University of Geneva in Switzerland, respectively first and last author of a paper entitled “In vivo magnetic resonance 31 P-Spectral Analysis With Neural Networks: 31P-SPAWNN”. This work was singled out because it demonstrated exemplary reproducible research practices; specifically, the authorsshared the source code for their proposed method (simulations, their deep learning network, and sample datasets).\nTo learn more, check outour recent MRM Highlights Q&A interview with Julien Songeon and Antoine Klauser.\n1. Why did you choose to share your code/data?\nThis approach naturally facilitates the demonstration presented in the manuscript, allowing interested readers to test the method themselves. Also, we enable others to validate and reproduce our results effectively.\nAlso, rather than adopting a competitive approach, we prefer, in this way, to promote collaboration in MR research, because a collaborative mindset can foster advancements and progress in the field. Ultimately, code sharing can assist other researchers who wish to pursue similar research directions. They can utilize our work as a foundation, potentially saving significant implementation and coding efforts. This accelerates the pace of research and encourages researchers to build upon existing knowledge rather than wasting resources on replication efforts.\n2. What is your lab or institutional policy on sharing research code and data?\nOur institution has a highly flexible policy regarding the sharing of code and data. While open science practices are strongly encouraged, there are no clear advantages for us nor mandatory constraints in place that require us to share .\n3. How do you think we might encourage researchers in the MRI community to contribute more open-source content along with their research papers?\nI believe that if code and data sharing were recognized to add clear value to a submitted manuscript, then authors would have a positive incentive to provide open-source content. In addition to evaluating factors such as innovation, soundness of methodology, quality of results, and clarity of demonstration, editors should acknowledge the inclusion of code and data as a valuable contribution to the scientific community.\n1. Your code repository is really well documented. What advice would you give to people preparing to share their first repository?\nWhen preparing to share a code repository, you need to think about the future users of your code and what information they will need to understand and use it effectively. The README file should explain the purpose of the code, and provide installation instructions, dependencies, and examples. The files could have comments within the code to clarify specific sections or functions. To help users navigate the repository and locate relevant files easily, I suggest organizing your code and files in a logical manner, using explicit naming. Finally, I would recommend including examples and/or tutorials, providing sample input data and scripts that demonstrate how to use your code. This can really help users to understand the expected workflow and apply the code to their own research.\n2. What questions did you ask yourselves while you were developing the code that would eventually be shared?\nIn the development phase, we aimed to create code that could be easily adapted and reused for different experiments or in different research scenarios.We also considered the computational complexity and performance of our code to ensure that it could handle large training datasets or complex computations effectively.Finally, we paid attention to the clarity and comprehensibility of our code. We aimed to write code that would be accessible to other researchers.\n3. How do you recommend that people use the project repository you shared?\nStart with the README file. Read the README file and try to rerun the command shown. it will provide an overview of the capability and usage of the code and model.Once you have a good understanding of the code, try to run it on your own data, adapting the number of points and the bandwidth and retraining a new model.Then, further adaptation could be implemented: you could add other metabolites, apply the method to other nuclei, explore other model architectures.\n4. Are there any other reproducible research habits that you didn’t use for this paper but might be interested in trying in the future?\nUsing tools likeDockerorSingularityto create reproducible and portable computing environments. This ensures that the code can be executed consistently across different systems and configurations without cumbersome installation or setup. In addition, keeping track of and archiving the successive versions of the code in a systematic manner allows researchers to easily track and manage changes to the code over time. This can be particularly useful when code is linked to a publication and extra data are required during the peer-review process. Also, we could try making automated pipelines that capture the entire workflow, from data acquisition to final results. Doing this simplifies the reproducibility process and reduces the potential for human error.\n'}, {'role': 'assistant', 'content': 'Keywords: 31P‐SPAWNN, neural networks, reproducibility, magnetic resonance spectroscopy\n\nStudy Achievements and Reproducibility Summary:\n\n1. The study introduced a new artificial intelligence framework, 31P‐SPAWNN, for analyzing 31P magnetic resonance spectra, demonstrating that it provides a fast and flexible alternative to traditional least-square fitting methods.\n2. Reproducibility was exemplary as the authors shared the source code, simulations, their deep learning network, and sample datasets, enabling other researchers to test, validate, and reproduce their results easily.\n3. Open sharing of the code and data was motivated by a commitment to collaborative advancement in MR research and the drive to facilitate incremental knowledge growth within the community without redundant effort.\n4. The authors suggest best practices for sharing research repositories, including comprehensive documentation, logical organization, and the use of tools like Docker or Singularity to improve the portability and consistency of the computational research environment, thereby enhancing reproducibility further.'}]
# List of dictionaries
if config['OPENAI_KEY']:
# THIS ASSUMES THAT THE OUTPUT WAS MADE AVAILABLE IN THE PREVIOUS CELL
messages.append({"role": "user", "content": f"Generate a comprehensive scoping review on the reproducibility of neuroimaging, incorporating insights from the 31 articles you summarized in our previous conversation. Provide examples and key findings to offer a thorough examination of the current state of reproducibility in neuroimaging research."})
response = client.chat.completions.create(
model="gpt-4-1106-preview",
messages= messages)
cur_response = {"role": "assistant", "content": response.choices[0].message.content}
messages.append(cur_response)
article = messages[-1]['content']
with open(get_output_dir('review_article_summary.txt'), 'w') as file:
file.write(article)
print(messages[-1]['content'])
else:
# Simply print the output
print('API key not found. An example output will be displayed from the previous results. \n')
gpt_txt = read_file(os.path.join(DATA_ROOT, "review1.txt"))
print(gpt_txt)API key not found. An example output will be displayed from the previous results.
**Scoping Review: Reproducibility in Neuroimaging Research**
**Introduction**
Reproducibility remains a cornerstone of scientific inquiry, its importance amplified in the complex and rapidly evolving field of neuroimaging. Studies leveraging magnetic resonance imaging (MRI), magnetic resonance spectroscopy (MRS), and other neuroimaging modalities are inherently data-intensive and computationally demanding. Ensuring reproducibility thus poses a unique set of challenges and necessitates the diligent application of methods that foster transparency, verification, and extension of research findings. This scoping review examines the current landscape of reproducible research practices in neuroimaging across various studies, drawing attention to common strategies, tools, and repositories used to achieve reproducible outcomes.
**Methods and Techniques for Ensuring Reproducibility**
Reproducibility in neuroimaging research is predicated upon the open sharing of comprehensive data sets, the code underlying data processing and analysis, and the complete details of the experimental setup. A review of recent neuroimaging research articles reveals several key themes central to facilitating reproducibility in the field:
1. **Open-Source Software and Code Sharing:** The widespread adoption of open-source software such as FSL-MRS for MRS analysis, hMRI for MRI analysis, and Dockerfiles for environment replication reveals a community trend towards accessible and transparent computational tools. Papers detailing deep learning approaches, such as DeepCEST for chemical exchange saturation transfer (CEST) MRI analysis, typically accompany methodological descriptions with shared code repositories and pre-trained models. These resources enable other scientists to test, apply, and enhance the computational techniques introduced.
2. **Data Sharing and Standardization:** The growing endorsement of data sharing is reflected in the number of studies providing access to raw imaging data, pre-processing pipelines, and post-analysis results. Efforts to standardize data formats, such as using NIfTI for spectroscopy data or adhering to Brain Imaging Data Structure (BIDS) for organizing complex datasets, are of increasing importance. Repositories like Zenodo, XNAT, and the Open Science Framework (OSF) serve as vital resources for housing and curating neuroimaging data.
3. **Integration of Hardware and Software:** Several studies such as those integrating RF coils with commercial field cameras for ultrahigh-field MRI or developing vendor-neutral imaging sequences for quantitative MRI exemplify the coupling of hardware advancements with software solutions. Sharing design files (e.g., CAD) and performance data ensures that hardware innovations can be reproduced and utilized by other researchers.
4. **Quality Assurance and Validation:** Implementing quality assurance (QA) measures is essential for the validation of neuroimaging findings. The introduction of tools for automated QA processes, as seen in the development of platforms like PreQual for DWI analysis, signifies an emphasis on ensuring the integrity and reliability of neuroimaging data.
**Discussion and Future Directions**
The components of reproducibility in neuroimaging research are multifaceted, integrating not just data and code, but also the analytical pipelines and hardware configurations. The shift towards comprehensive sharing is motivated both by a scientific ethic of transparency and the practical need for rigorous validation of complex methodologies. As computational resources become more sophisticated, there are increasing calls for the incorporation of standardized descriptors, metadata, and data formats within the community.
Reproducibility is also bolstered by interactive documentation and tools such as Jupyter Notebooks, allowing for dynamic presentation and hands-on engagement with data and methods. Platforms incorporating such interactive elements are being utilized with greater frequency, providing real-time demonstration of analysis techniques and enabling peer-led validation.
The rise of machine learning and artificial intelligence in neuroimaging demands meticulous scrutiny for reproducibility. Studies implementing these technologies are beginning to pair their methodological descriptions with the open-source code, trained models, and simulation tools that underpin their algorithms.
**Conclusion**
The progress towards reproducibility in neuroimaging research points to a distinct cultural shift in the scientific community. The move towards open-access publishing, code-sharing platforms, and data repositories reflects a concerted effort to uphold the reproducibility of complex imaging studies. The initiatives and tools identified in this review serve as a blueprint for future studies to replicate successful practices, safeguards against bias, and accelerate neuroscientific discovery. As neuroimaging research continues to advance, upholding the principles of reproducibility will be essential to maintaining the integrity and translational potential of its findings.
if config['OPENAI_KEY']:
# messages variable should be in scope. Please see previous cells.
# Fetch this from the saved data
input_files = [os.path.join(DATA_ROOT, "gpt_insights_parsed_nov23", f) for f in os.listdir(directory_path) if f.endswith('.txt')]
if not os.path.exists(get_output_dir("summary")):
os.mkdir(get_output_dir("summary"))
summary_files = [os.path.join(get_output_dir("summary"), f) for f in os.listdir(directory_path) if f.endswith('.txt')]
large_summary = ''
it = 0
for msg in messages:
if msg['role'] == "assistant":
large_summary += "\n" + msg['content'] + "\n ---------------------------- \n"
write_file(summary_files[it],msg['content'])
it = it +1
write_file(get_output_dir('gpt_combined_summaries.txt'),large_summary)
else:
print('API key not found. 3 example outputs will be displayed. \n')
gpt_outputs = os.path.join(DATA_ROOT, "gpt_chat.pkl")
messages = read_pickle(gpt_outputs)
it = 0
for msg in messages:
if msg['role'] == "assistant":
print(msg['content'] + '\n')
it+=1
if it == 3:
break
API key not found. 3 example outputs will be displayed.
Keywords: 31P‐SPAWNN, neural networks, reproducibility, magnetic resonance spectroscopy
Study Achievements and Reproducibility Summary:
1. The study introduced a new artificial intelligence framework, 31P‐SPAWNN, for analyzing 31P magnetic resonance spectra, demonstrating that it provides a fast and flexible alternative to traditional least-square fitting methods.
2. Reproducibility was exemplary as the authors shared the source code, simulations, their deep learning network, and sample datasets, enabling other researchers to test, validate, and reproduce their results easily.
3. Open sharing of the code and data was motivated by a commitment to collaborative advancement in MR research and the drive to facilitate incremental knowledge growth within the community without redundant effort.
4. The authors suggest best practices for sharing research repositories, including comprehensive documentation, logical organization, and the use of tools like Docker or Singularity to improve the portability and consistency of the computational research environment, thereby enhancing reproducibility further.
Keywords: Cystic Fibrosis, Hyperpolarized 129Xe MRI, Dynamic 19F MRI, Ventilation Abnormalities
Study Achievements and Reproducibility Summary:
1. The study achieved a quantitative comparison between dynamic 19F MRI and single breath hyperpolarized 129Xe MRI for detecting ventilation abnormalities in subjects with mild CF lung disease, contributing valuable insights into the efficacy of different imaging modalities for CF assessment.
2. Reproducibility was enhanced by sharing the code and data that reproduce several figures, allowing others in the research community to validate and potentially extend the findings.
3. The decision to share came during the submission process, and recognizing the benefits, including feedback and collaboration, the team has been motivated to continue sharing for future publications.
4. The researchers advocate for culture change within the MRI community where data sharing becomes an expectation and plan to explore additional reproducibility tools, such as explanatory videos, to further disseminate their research methods and findings.
Keywords: Dynamic MRI, Non-gated Acquisition, Large-scale Volumetric Imaging, Pulmonary Imaging
Study Achievements and Reproducibility Summary:
1. The study achieved the development of a framework capable of reconstructing large-scale volumetric dynamic MRI images from continuous, non-gated acquisitions, with successful applications demonstrated in pulmonary and DCE imaging.
2. Reproducibility was prioritized through the sharing of code and an interactive Google Colab notebook demo, which allows for easy visualization and manipulation of the reconstruction process.
3. The culture of sharing within UC Berkeley and the research team directly influenced the early decision to share code/data, with the practice of version controlling, documentation, and unit tests facilitating the sharing process.
4. Encouraging open-source contributions in the MRI community via showcases, hands-on sessions, and interviews has proven effective, and the authors intend to continue these efforts while exploring additional platforms like Zenodo for data hosting.
if config['OPENAI_KEY']:
# Output path will be checked for the summaries
summary_files = [os.path.join(get_output_dir("summary", f)) for f in os.listdir(directory_path) if f.endswith('.txt')]
# List to store summaries
summaries = []
# Iterate through each txt file in the directory
for filename in summary_files:
content = read_file(filename)
summaries.append(content)
# Create a Pandas DataFrame with a column named 'summary'
df = pd.DataFrame({'summary': summaries})
embedding_model = "text-embedding-ada-002"
embedding_encoding = "cl100k_base" # this the encoding for text-embedding-ada-002
max_tokens = 8000
top_n = 1000
encoding = tiktoken.get_encoding(embedding_encoding)
# omit reviews that are too long to embed
df["n_tokens"] = df.summary.apply(lambda x: len(encoding.encode(x)))
df = df[df.n_tokens <= max_tokens].tail(top_n)
# Call GPT for embedding
df["embedding"] = df.summary.apply(lambda x: get_embedding(x, model='text-embedding-ada-002'))
df.to_csv(get_output_dir('gpt_summary_embedding.csv'), index=False)
else:
# Read embedding data that has been saved
df = pd.read_csv(os.path.join(DATA_ROOT,'gpt_summary_embedding_nov23.csv'))
print(df.head()) summary n_tokens \
0 Keywords: 31P‐SPAWNN, neural networks, reprodu... 201
1 Keywords: Cystic Fibrosis, Hyperpolarized 129X... 203
2 Keywords: Dynamic MRI, Non-gated Acquisition, ... 190
3 Keywords: Real-Time MRI, Speech Production, 3D... 215
4 Keywords: Inhomogeneous Magnetization Transfer... 195
embedding
0 [-0.012139617465436459, 0.015481614507734776, ...
1 [-0.0038842952344566584, 0.019686615094542503,...
2 [-0.025156814604997635, 0.017279978841543198, ...
3 [-0.028323667123913765, 0.013008967973291874, ...
4 [-0.038999006152153015, 0.013461689464747906, ...
# Get embeddings as matrix from the dataframe
matrix = df.embedding.apply(eval).to_list()
# Use T-distributed Stochastic Neighbor Embedding to create joint probabilities.
tsne = TSNE(n_components=2, perplexity=15, random_state=42, init='random', learning_rate=200)
vis_dims = tsne.fit_transform(np.array(matrix))
x = [x for x,y in vis_dims]
y = [y for x,y in vis_dims]
fig = go.Figure()
# Scatter plot for UMAP in 2D
scatter_2d = go.Scatter(
x=x,
y=y,
mode='markers',
# Color w.r.t. summary length
marker = dict(color =[len(sm) for sm in df.summary],
size=9,
opacity=0.9,
colorbar=dict(thickness=10,x=1.1, title="N chars")),
customdata= [sm.split('\n')[0] for sm in df.summary],
hovertemplate='%{customdata}',
visible = True,
name=''
)
fig.add_trace(scatter_2d)
fig.update_layout(yaxis={'visible': False, 'showticklabels': False})
fig.update_layout(xaxis={'visible': False, 'showticklabels': False})
# Update layout
fig.update_layout(title = 't-SNE based visualization of the summary embeddings',
height = 800,
width = 800,
template = 'plotly_dark',
hovermode='closest',
showlegend = True)
fig.show()Loading...
import os
import requests
DATA_ROOT = "../data/repro-mri-scoping/repro_mri_scoping/gpt_summary_nov23"
# Function to make Semantic Scholar API call and extract title and authors
def get_paper_info(paper_id):
api_url = f"https://api.semanticscholar.org/graph/v1/paper/{paper_id}?fields=url,year,authors,title"
response = requests.get(api_url)
if response.status_code == 200:
data = response.json()
title = data.get('title', 'Title Not Available')
year = data.get('year', 'Year Not Available')
authors = ', '.join(author.get('name', 'Unknown Author') for author in data.get('authors', []))
return title, authors, year
else:
print(f"Failed to fetch data for paper_id: {paper_id}")
return None, None, None
# Directory containing txt files
directory = DATA_ROOT
# Initialize an empty list to store data
combined_data = []
# Loop through each txt file in the directory
it =1
for filename in os.listdir(directory):
if filename.endswith(".txt"):
paper_id = os.path.splitext(filename)[0]
file_path = os.path.join(directory, filename)
# Read content of the txt file
with open(file_path, 'r', encoding='utf-8') as file:
content = file.read().strip()
# Make Semantic Scholar API call to get paper title and authors
title, authors, year = get_paper_info(paper_id)
# Append data to the list
combined_data.append(f"\nPaper number: {it}\nYear: {year}\nTitle: {title}\nAuthors: {authors}\n\n{content}\n\n ---------------------------------------------------")
it = it+1
# Write the combined data to a text file
with open(os.path.join(get_output_dir('combined_papers_insight.txt')), 'w', encoding='utf-8') as output_file:
output_file.writelines(combined_data)
Failed to fetch data for paper_id: 4a3c2a337873727ae603e3d6d22616ec8c44d983
Failed to fetch data for paper_id: 7314d44b2a2a6ef644ebbd9c9dfe6e0f750cd548
Failed to fetch data for paper_id: 0ed3793f85a966628b0a4ea55782c1b86f264c94
Failed to fetch data for paper_id: eedde84af13da6da7a3d4dc79da7092a176fb4de
Failed to fetch data for paper_id: 904e52b49bc3edd15750529af3e8a74ca53c5086
Failed to fetch data for paper_id: 74996eb542012e1c95663f6e7f6b386bcfac3347
Failed to fetch data for paper_id: abf8a8cdf69f4acad310db77a20a82bdb054e23b
Failed to fetch data for paper_id: ac86f8a4db699af7f4a08c762f45fff4bcc66e9b
Failed to fetch data for paper_id: a78ade2cdcc92914070dff9ea7ffd742a099c6a5
Failed to fetch data for paper_id: b39090dca7c82afb41a3b9fa077809ba03158322
Failed to fetch data for paper_id: 7b7e5effcb26e7dba7d664a8b00ef8990530bb03
Failed to fetch data for paper_id: ea834c55ff96559761ba9e8c1dc37cf6469491ee
Failed to fetch data for paper_id: 72f9a757d1ed8086ff98d61bec294e830d4df971
Failed to fetch data for paper_id: 53426a1aabd39dd46040e4757048afc3c65573d6
Failed to fetch data for paper_id: 92956b8a300266a0fa5557cf6ea5a925f1e2fca8
Failed to fetch data for paper_id: 7da6204035e97bdae2e748066c9b7148d77a755c
Failed to fetch data for paper_id: d9ae0cb83af34b23d6d196a49f29572ed177d9dd
Failed to fetch data for paper_id: c8ba4550cc1d451f3fca5a976b0686f806d8b233
Failed to fetch data for paper_id: fed7cb7ca6c353b16db9d9d02fdecb4ef56cc751
Failed to fetch data for paper_id: 67244f2062988f21b4032ff2795b4fce436cbc35
Failed to fetch data for paper_id: dbd7bd77f8b37489b8fabd9ee16782f3f19bccfb
Failed to fetch data for paper_id: 75a2717267d724233006f1e36fea6ffc133ec695
Failed to fetch data for paper_id: 55cca9c2910abd0eff0200af23191483be93ccf9
Failed to fetch data for paper_id: bd1967c25311040f309465453e39ec8030ae011e
Failed to fetch data for paper_id: 02713a1d17977fdb013f20c804274051eab65998
Failed to fetch data for paper_id: c1d4563f657b32ae5680cc00d3e8f754c17c7d97
Failed to fetch data for paper_id: 2786c45adc34d59b007cf275d7578f05d5f90f28