appengine aetest的行为不一致

huangapple go评论79阅读模式
英文:

Inconsistent behaviour from appengine aetest

问题

我正在尝试测试一个函数,该函数应该从数据存储中获取某种类型的所有对象。

在测试中,似乎我必须插入一个延迟以使查询找到所有保存的项。

下面的代码是一个可复现的示例。当注释掉延迟行时,第一个日志跟踪是从第二个日志跟踪开始的,注意长度:0 和长度:3。

我认为这是一个最终一致性问题,在生产环境中,如果我持久化了几个对象并立即查询它们,就会出现这个问题。但在生产环境中,这些项早就被持久化了。由于这种情况只出现在我的测试中,我是否应该做些什么来强制数据存储等待完全保存项后再继续?

我尝试将测试保存的内容包装在一个事务中,但是得到了一个"Only ancestor queries are allowed inside transactions"的错误。

type Thing struct {
  Str1 string
  Str2 string
}

func (thing Thing) Save(c appengine.Context) error {
  k := datastore.NewKey(c, "Thing", thing.Str1 + "_" + thing.Str2, 0, nil)
  if _, err := datastore.Put(c, k, &thing); err != nil {
    return err
  }
  return nil
}

func GetThings(c appengine.Context) ([]Thing, error) {
  var things []Thing
  q := datastore.NewQuery("Thing").
    Filter("Str1=", "thing")
  _, err := q.GetAll(c, &things)
  if err != nil {
    return nil, err
  }
  return things, nil
}

func TestGetThings(t *testing.T) {
  c, _ := aetest.NewContext(nil)
  defer c.Close()
  thing1 := Thing{"thing", "1"}
  thing2 := Thing{"thing", "2"}
  thing3 := Thing{"thing", "3"}
  thing1.Save(c)
  thing2.Save(c)
  thing3.Save(c)

  // time.Sleep(2000 * time.Millisecond)

  things, err := GetThings(c)
  if err != nil {
    t.Fatal(err)
  }
  t.Log("length:" + strconv.Itoa(len(things)))
}

当延迟被注释时的日志

C:\Users\XXXX>goapp test thing -test.v

2014/09/23 21:24:05 appengine: not running under devappserver2; using some default configuration

=== RUN TestGetThings

INFO 2014-09-23 21:24:07,328 devappserver2.py:725] Skipping SDK update check.

WARNING 2014-09-23 21:24:07,328 devappserver2.py:741] DEFAULT_VERSION_HOSTNAME will not be set correctly with --port=0

WARNING 2014-09-23 21:24:07,351 api_server.py:383] Could not initialize images API; you are likely missing the Python "PIL" module.

INFO 2014-09-23 21:24:07,365 api_server.py:171] Starting API server at: http://localhost:50153

INFO 2014-09-23 21:24:07,371 dispatcher.py:183] Starting module "default" running at: http://localhost:50154

INFO 2014-09-23 21:24:07,377 admin_server.py:117] Starting admin server at: http://localhost:50155

INFO 2014-09-23 21:24:08,378 api_server.py:583] Applying all pending transactions and saving the datastore

INFO 2014-09-23 21:24:08,388 api_server.py:586] Saving search indexes
--- PASS: TestGetThings (4.60 seconds)

thing_test.go:87: length:0

PASS

ok thing 4.729s

当延迟未被注释时的日志

C:\Users\XXXX>goapp test thing -test.v

2014/09/23 21:24:28 appengine: not running under devappserver2; using some default configuration

=== RUN TestGetThings

INFO 2014-09-23 21:24:31,124 devappserver2.py:725] Skipping SDK update check.

WARNING 2014-09-23 21:24:31,124 devappserver2.py:741] DEFAULT_VERSION_HOSTNAME will not be set correctly with --port=0

WARNING 2014-09-23 21:24:31,148 api_server.py:383] Could not initialize images API; you are likely missing the Python "PIL" module.

INFO 2014-09-23 21:24:31,164 api_server.py:171] Starting API server at: http://localhost:50191

INFO 2014-09-23 21:24:31,171 dispatcher.py:183] Starting module "default" running at: http://localhost:50192

INFO 2014-09-23 21:24:31,176 admin_server.py:117] Starting admin server at: http://localhost:50193

INFO 2014-09-23 21:24:34,176 api_server.py:583] Applying all pending transactions and saving the datastore

INFO 2014-09-23 21:24:34,176 api_server.py:586] Saving search indexes
--- PASS: TestGetThings (6.83 seconds)

thing_test.go:87: length:3

PASS

ok thing 6.987s

英文:

I'm trying to test a function that should fetch all the objects of a certain kind from the datastore.

Within the test it appears I have to insert a sleep to make the query find all the saved items.

The code below is a reproducible example. The first log trace is from when the sleep line is commented out, and the second when the sleep is uncommented. Note the length: 0 and length: 3

I'm assuming this is an eventual consistency issue, which would arise in production if I persisted several objects and immediately queried for them. But in production, these items were persisted long beforehand. Since this situation only arises for my test, is there something I'm meant to be doing to force the datastore to wait till it has fully saved the items before continuing?

I tried wrapping the test saves in a transaction, but got an "Only ancestor queries are allowed inside transactions" error.

type Thing struct {
  Str1 string
  Str2 string
}

func (thing Thing) Save(c appengine.Context) error {
  k := datastore.NewKey(c, "Thing", thing.Str1 + "_" + thing.Str2, 0, nil)
  if _, err := datastore.Put(c, k, &thing); err != nil {
    return err
  }
  return nil
}

func GetThings(c appengine.Context) ([]Thing, error) {
  var things []Thing
  q := datastore.NewQuery("Thing").
	  Filter("Str1=", "thing")
  _, err := q.GetAll(c, &things)
  if err != nil {
    return nil, err
  }
  return things, nil
}



func TestGetThings(t *testing.T) {
  c, _ := aetest.NewContext(nil)
  defer c.Close()
  thing1 := Thing{"thing", "1"}
  thing2 := Thing{"thing", "2"}
  thing3 := Thing{"thing", "3"}
  thing1.Save(c)
  thing2.Save(c)
  thing3.Save(c)

//	time.Sleep(2000 * time.Millisecond)

  things, err := GetThings(c)
  if err != nil {
	t.Fatal(err)
  }
  t.Log("length:" + strconv.Itoa(len(things)))
}

Logs when sleep commented out

C:\Users\XXXX>goapp test thing -test.v

2014/09/23 21:24:05 appengine: not running under devappserver2; using some default configuration

=== RUN TestGetThings

INFO 2014-09-23 21:24:07,328 devappserver2.py:725] Skipping SDK update check.

WARNING 2014-09-23 21:24:07,328 devappserver2.py:741] DEFAULT_VERSION_HOSTNAME will not be set correctly with --port=0

WARNING 2014-09-23 21:24:07,351 api_server.py:383] Could not initialize images API; you are likely missing the Python "PIL" module.

INFO 2014-09-23 21:24:07,365 api_server.py:171] Starting API server at: http://localhost:50153

INFO 2014-09-23 21:24:07,371 dispatcher.py:183] Starting module "default" running at: http://localhost:50154

INFO 2014-09-23 21:24:07,377 admin_server.py:117] Starting admin server at: http://localhost:50155

INFO 2014-09-23 21:24:08,378 api_server.py:583] Applying all pending transactions and saving the datastore

INFO 2014-09-23 21:24:08,388 api_server.py:586] Saving search indexes
--- PASS: TestGetThings (4.60 seconds)

thing_test.go:87: length:0

PASS

ok thing 4.729s

Logs when sleep not commented out

C:\Users\XXXX>goapp test thing -test.v

2014/09/23 21:24:28 appengine: not running under devappserver2; using some default configuration

=== RUN TestGetThings

INFO 2014-09-23 21:24:31,124 devappserver2.py:725] Skipping SDK update check.

WARNING 2014-09-23 21:24:31,124 devappserver2.py:741] DEFAULT_VERSION_HOSTNAME will not be set correctly with --port=0

WARNING 2014-09-23 21:24:31,148 api_server.py:383] Could not initialize images API; you are likely missing the Python "PIL" module.

INFO 2014-09-23 21:24:31,164 api_server.py:171] Starting API server at: http://localhost:50191

INFO 2014-09-23 21:24:31,171 dispatcher.py:183] Starting module "default" running at: http://localhost:50192

INFO 2014-09-23 21:24:31,176 admin_server.py:117] Starting admin server at: http://localhost:50193

INFO 2014-09-23 21:24:34,176 api_server.py:583] Applying all pending transactions and saving the datastore

INFO 2014-09-23 21:24:34,176 api_server.py:586] Saving search indexes
--- PASS: TestGetThings (6.83 seconds)

thing_test.go:87: length:3

PASS

ok thing 6.987s

答案1

得分: 3

为了回答自己的问题,解决方案是在创建aetest.NewContext时使用Options参数。

c, _ := aetest.NewContext(&aetest.Options{"", true})

Options结构体中的第二个参数是一个名为StronglyConsistentDatastore的布尔值。

英文:

To answer my own question, the solution is to use the Options parameter when creating the aetest.NewContext.

c, _ := aetest.NewContext(&aetest.Options{"", true})

The second item in the Options struct is a boolean named StronglyConsistentDatastore.

答案2

得分: 0

你遇到了最终一致性的情况。你可以在上面的链接中阅读相关内容,或者在GoLang概述中了解更多。你需要做出选择,是更早获取数据还是写入速度更慢。如果你在放置数据后立即需要访问它,你需要进行祖先查询,这会减慢写入速度,但可以立即访问数据。祖先查询还有其他限制(如事务),但对于你目前的问题,它可以解决你的问题。

英文:

You are running into Eventual Consistency situation. You can read on that in the prev link or in the GoLang Overview. You need to make a choice, is getting the data sooner worth having slower writes. If you need your data right after you put it you need to do Ancestor Queries which will slow down writes but will make your data available immediately after. There are other constraints with Ancestors (like transactions) but for your immediate concern it will solve your issue.

huangapple
  • 本文由 发表于 2014年9月24日 04:50:18
  • 转载请务必保留本文链接:https://go.coder-hub.com/26004506.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定