Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

关于libcity news中提到的几篇论文的实验结果差距 #367

Closed
Transformer0714 opened this issue Oct 11, 2023 · 2 comments
Closed

Comments

@Transformer0714
Copy link

同一个模型,三篇论文报告的实验结果为什么差距这么大,请问都是在libcity库的基础上开发的吗?
例如STTN在PEMS08上一小时的流量预测:
MAE 16.98, MAPE 12.87 RMSE 27.75(Towards Efficient and Comprehensive Urban Spatial-Temporal
Prediction: A Unified Library and Performance Benchmark)
MAE 15.48, MAPE 10.34 RMSE 24.96(PDFormer)
MAE 16.10, MAPE 11.28 RMSE 24.70(Unified Data Management and Comprehensive Performance
Evaluation for Urban Spatial-Temporal Prediction)

@aptx1231
Copy link
Member

aptx1231 commented Oct 11, 2023

同学你好。

MAE 15.48, MAPE 10.34 RMSE 24.96(PDFormer)这个是12步平均的结果,而且是6:2:2的数据划分。这个是遵循过去文章的规范,对于这个数据集都是6:2:2的划分方案。

MAE 16.10, MAPE 11.28 RMSE 24.70(Unified Data Management and Comprehensive Performance
Evaluation for Urban Spatial-Temporal Prediction) 这个是12步平均的结果,而且是7:1:2的数据划分。这个是因为这个文章比较了很多数据,采用的通义的7:1:2, 是DCRNN论文开创的划分方案。

最后,MAE 16.98, MAPE 12.87 RMSE 27.75(Towards Efficient and Comprehensive Urban Spatial-Temporal
Prediction: A Unified Library and Performance Benchmark) 这个首先例子不太对,因为这个文章提供的不是12步平均,因为arxiv文章的空间不受限制,所以给的第3,6,12步的结果,对于多步预测,第12步的结果肯定比12步平均差的呀,所以这个结果比较大。

如果不太理解平均和单步,可以看这个:#359

最后,也是为了避免误解,Towards Efficient and Comprehensive Urban Spatial-Temporal
Prediction: A Unified Library and Performance Benchmark这个arxiv目前改成LibCity: A Unified Library Towards Efficient and Comprehensive Urban Spatial-Temporal Prediction了,作为libcity短文的扩展版本,不再放实验结果了。

@Transformer0714
Copy link
Author

好的,了解了,非常感谢

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants