------------------------- METAREVIEW ------------------------ I will recommend accepting this paper as there is clear consensus. ----------------------- REVIEW 1 --------------------- SUBMISSION: 2628 TITLE: Locker: Locally Constrained Self-Attentive Sequential Recommendation AUTHORS: Zhankui He, Handong Zhao, Zhe Lin, Zhaowen Wang, Ajinkya Kale and Julian Mcauley ----------- Relevance to CIKM ----------- SCORE: 4 (good) ----------- Originality of the Work ----------- SCORE: 4 (good) ----------- Technical Soundness ----------- SCORE: 3 (fair) ----------- Quality of Presentation ----------- SCORE: 5 (excellent) ----------- Impact of Ideas or Results ----------- SCORE: 4 (good) ----------- Adequacy of Citations ----------- SCORE: 4 (good) ----------- Reproducibility of Methods ----------- SCORE: 3 (fair) ----------- Reviewer's confidence ----------- SCORE: 4 (good) ----------- Overall Recommendation ----------- SCORE: 1 (Weak Accept) ----------- Nominate for best paper ----------- SELECTION: no ----------- Comments to the Author(s) ----------- The paper empirically shows that the existing self-attentive methods for sequential recommendation cannot effectively learn short-term user dynamics. The proposed solution is to combine modules that focus on local items in replace of some attention heads. The author further shows that reweighting the attentional weights of the encoders using a learned user-aware masking score predictor can achieve a better performance boost without much computational overhead. The motivation of the paper is clear and the supporting experiments are comprehensive and convincing. In general, the paper is easy to read and could have a moderately significant impact on the area of sequential recommendation. Some minor questions: 1. Figure 1, does “conv” mean convolution kernel mentioned in sections 3.2.2? 2. Figure 2-d, the attention map shows that the self-attention models are capable of assigning larger weights to local items than further items in history, which potentially contradicts the statement that these models are not capable. One may argue that the observation of unsatisfactory encoding of short-term items is the result of insufficient training. However, as mentioned by the user, the data is usually insufficient. 3. Line 314, “doos” 4. The adaptive weights also require additional storage for extra parameters, it would be more insightful if giving a corresponding analysis on space cost. 5. The controlled experiments on the local constraints are an insightful approach but potentially require a more careful design. For example, to observe how much short-term items contribute to the final prediction compared to long-term items. A more comprehensive experiment would also include a complementary comparison for long-term items (i.e. x% long-term items + a% random short-term items). ----------- Summary ----------- The motivation of the paper is clear and the supporting experiments are comprehensive and convincing. In general, the paper is easy to read and could have a moderately significant impact on the area of sequential recommendation. ----------------------- REVIEW 2 --------------------- SUBMISSION: 2628 TITLE: Locker: Locally Constrained Self-Attentive Sequential Recommendation AUTHORS: Zhankui He, Handong Zhao, Zhe Lin, Zhaowen Wang, Ajinkya Kale and Julian Mcauley ----------- Relevance to CIKM ----------- SCORE: 4 (good) ----------- Originality of the Work ----------- SCORE: 3 (fair) ----------- Technical Soundness ----------- SCORE: 4 (good) ----------- Quality of Presentation ----------- SCORE: 4 (good) ----------- Impact of Ideas or Results ----------- SCORE: 4 (good) ----------- Adequacy of Citations ----------- SCORE: 4 (good) ----------- Reproducibility of Methods ----------- SCORE: 4 (good) ----------- Reviewer's confidence ----------- SCORE: 4 (good) ----------- Overall Recommendation ----------- SCORE: 1 (Weak Accept) ----------- Nominate for best paper ----------- SELECTION: no ----------- Comments to the Author(s) ----------- This paper studies if the current self-attention approach for recommendation can adapt to locally structure effectively. Motivated by the analysis, the authors also proposed a locally constrained framework which can be applied on top of existing self-attention-based recommendation approaches. Experiments on real-world datasets demonstrate the proposed framework yields significant performance gains on average. Strength: - The topic is relevant to CIKM community - The proposed method is also well-motivated, flexible and practical - Extensive experiments were conducted to demonstrate the effectiveness of the proposed framework Weakness: - A small caveat with the analysis - in the current set up, the authors attempted to generate pseudo noisy training instances to disturb the model; how would the model perform if it's trained on clean datasets but evaluated on these pseudo sequences? - Following the above, how the model would perform corresponding to the noisy pseudo sequences after applying the proposed local constraints - Typos/presentation mistakes in line 420, page 4: "(2) Presumably because RNNs encode actions …", there is another "(2)" following this sentence, the quotation mark in 'one-by-one' is also not rendered correctly ----------- Summary ----------- The paper is well-situated in the short paper track and interesting to read. Although the messages can be further sharpened and some improvements can be done in the analysis, I feel the paper was generally executed in a good shape and could shed some light to the recommender system community. ----------------------- REVIEW 3 --------------------- SUBMISSION: 2628 TITLE: Locker: Locally Constrained Self-Attentive Sequential Recommendation AUTHORS: Zhankui He, Handong Zhao, Zhe Lin, Zhaowen Wang, Ajinkya Kale and Julian Mcauley ----------- Relevance to CIKM ----------- SCORE: 5 (excellent) ----------- Originality of the Work ----------- SCORE: 3 (fair) ----------- Technical Soundness ----------- SCORE: 3 (fair) ----------- Quality of Presentation ----------- SCORE: 4 (good) ----------- Impact of Ideas or Results ----------- SCORE: 3 (fair) ----------- Adequacy of Citations ----------- SCORE: 4 (good) ----------- Reproducibility of Methods ----------- SCORE: 4 (good) ----------- Reviewer's confidence ----------- SCORE: 4 (good) ----------- Overall Recommendation ----------- SCORE: 1 (Weak Accept) ----------- Nominate for best paper ----------- SELECTION: no ----------- Comments to the Author(s) ----------- This paper describes an approach to next item recommendation. It targets the problem of lack of local information in the general attention mechanism. To this end, several approaches to capture local information , before feeding it to the attention mechanism, are proposed. They are all shown to be useful compared to the baseline models. The idea of incorporating local information in attention is interesting. However, this method is not new. The ways to capture local information are also quite standard now. So, the technical contribution is limited. It is however interesting to see that the idea is useful for recommendation. On the experiments, the results show that local information is useful, especially with the adaptive predictor and CNN. Looking at the values, however, one can see that the final performance is still very low, for example, NDCG@20 of less than 0.05. Do these values meaningful in practice? Is the method practically useful? I tend to say "not really". The performance is certainly dependent on the dataset. However, this level of performance may also mean that the general approaches (including the ones proposed in this paper) are far from useable in practice. ----------- Summary ----------- - reasonable approach that is based on known architecture. - improvements in the experiments. - however, the performance is generally very low (not truly significant in practice)