@Test public void useTransactionTest() throws Exception { // Performing admin operations to create dataset instance // keyValueTable is a system dataset module Id.DatasetInstance myTableInstance = Id.DatasetInstance.from(namespace, "myTable"); dsFramework.addInstance("keyValueTable", myTableInstance, DatasetProperties.EMPTY); Assert.assertTrue(feedManager.createFeed(FEED1)); try { Cancellable cancellable = notificationService.subscribe( FEED1, new NotificationHandler<String>() { private int received = 0; @Override public Type getNotificationType() { return String.class; } @Override public void received( final String notification, NotificationContext notificationContext) { notificationContext.execute( new TxRunnable() { @Override public void run(DatasetContext context) throws Exception { KeyValueTable table = context.getDataset("myTable"); table.write("foo", String.format("%s-%d", notification, received++)); } }, TxRetryPolicy.maxRetries(5)); } }); TimeUnit.SECONDS.sleep(2); try { notificationService.publish(FEED1, "foobar"); // Waiting for the subscriber to receive that notification TimeUnit.SECONDS.sleep(2); KeyValueTable table = dsFramework.getDataset(myTableInstance, DatasetDefinition.NO_ARGUMENTS, null); Assert.assertNotNull(table); Transaction tx1 = txManager.startShort(100); table.startTx(tx1); Assert.assertEquals("foobar-0", Bytes.toString(table.read("foo"))); Assert.assertTrue(table.commitTx()); txManager.canCommit(tx1, table.getTxChanges()); txManager.commit(tx1); table.postTxCommit(); } finally { cancellable.cancel(); } } finally { dsFramework.deleteInstance(myTableInstance); feedManager.deleteFeed(FEED1); } }
@BeforeClass public static void setupBeforeClass() throws Exception { testUtil = new HBaseTestingUtility(); testUtil.startMiniCluster(); conf = testUtil.getConfiguration(); hBaseAdmin = testUtil.getHBaseAdmin(); txStateStorage = new InMemoryTransactionStateStorage(); txManager = new TransactionManager(conf, txStateStorage, new TxMetricsCollector()); txManager.startAndWait(); }
@AfterClass public static void afterClass() throws Exception { deleteNamespaces(); streamService.stopAndWait(); appFabricServer.stopAndWait(); metricsCollectionService.stopAndWait(); metricsService.stopAndWait(); datasetService.stopAndWait(); dsOpService.stopAndWait(); txManager.stopAndWait(); }
public static void startServices(Injector injector) throws Exception { notificationService = injector.getInstance(NotificationService.class); notificationService.startAndWait(); feedManager = injector.getInstance(NotificationFeedManager.class); dsFramework = injector.getInstance(DatasetFramework.class); txManager = injector.getInstance(TransactionManager.class); txManager.startAndWait(); dsOpService = injector.getInstance(DatasetOpExecutor.class); dsOpService.startAndWait(); datasetService = injector.getInstance(DatasetService.class); datasetService.startAndWait(); }
@BeforeClass public static void beforeClass() throws Throwable { CConfiguration conf = CConfiguration.create(); conf.set(Constants.AppFabric.SERVER_ADDRESS, hostname); conf.set(Constants.CFG_LOCAL_DATA_DIR, tmpFolder.newFolder("data").getAbsolutePath()); conf.set(Constants.AppFabric.OUTPUT_DIR, System.getProperty("java.io.tmpdir")); conf.set(Constants.AppFabric.TEMP_DIR, System.getProperty("java.io.tmpdir")); conf.setBoolean(Constants.Scheduler.SCHEDULERS_LAZY_START, true); conf.set( Constants.AppFabric.APP_TEMPLATE_DIR, tmpFolder.newFolder("templates").getAbsolutePath()); conf.setBoolean(Constants.Dangerous.UNRECOVERABLE_RESET, true); DirUtils.mkdirs(new File(conf.get(Constants.AppFabric.APP_TEMPLATE_DIR))); DirUtils.mkdirs(new File(conf.get(Constants.AppFabric.APP_TEMPLATE_PLUGIN_DIR))); injector = Guice.createInjector(new AppFabricTestModule(conf)); txManager = injector.getInstance(TransactionManager.class); txManager.startAndWait(); dsOpService = injector.getInstance(DatasetOpExecutor.class); dsOpService.startAndWait(); datasetService = injector.getInstance(DatasetService.class); datasetService.startAndWait(); appFabricServer = injector.getInstance(AppFabricServer.class); appFabricServer.startAndWait(); DiscoveryServiceClient discoveryClient = injector.getInstance(DiscoveryServiceClient.class); ServiceDiscovered appFabricHttpDiscovered = discoveryClient.discover(Constants.Service.APP_FABRIC_HTTP); EndpointStrategy endpointStrategy = new RandomEndpointStrategy(appFabricHttpDiscovered); port = endpointStrategy.pick(1, TimeUnit.SECONDS).getSocketAddress().getPort(); txClient = injector.getInstance(TransactionSystemClient.class); metricsCollectionService = injector.getInstance(MetricsCollectionService.class); metricsCollectionService.startAndWait(); metricsService = injector.getInstance(MetricsQueryService.class); metricsService.startAndWait(); streamService = injector.getInstance(StreamService.class); streamService.startAndWait(); serviceStore = injector.getInstance(ServiceStore.class); serviceStore.startAndWait(); streamAdmin = injector.getInstance(StreamAdmin.class); locationFactory = getInjector().getInstance(LocationFactory.class); adapterDir = new File(conf.get(Constants.AppFabric.APP_TEMPLATE_DIR)); createNamespaces(); }
/** * Test full stack serialization for a TransactionManager migrating from DefaultSnapshotCodec to * SnapshotCodecV3. */ @Test public void testV2ToTephraV3Migration() throws Exception { File testDir = tmpDir.newFolder("testV2ToTephraV3Migration"); Configuration conf = new Configuration(); conf.setStrings( TxConstants.Persist.CFG_TX_SNAPHOT_CODEC_CLASSES, SnapshotCodecV1.class.getName(), SnapshotCodecV2.class.getName()); conf.set(TxConstants.Manager.CFG_TX_SNAPSHOT_LOCAL_DIR, testDir.getAbsolutePath()); Injector injector = Guice.createInjector( new ConfigModule(conf), new DiscoveryModules().getSingleNodeModules(), new TransactionModules().getSingleNodeModules()); TransactionManager txManager = injector.getInstance(TransactionManager.class); txManager.startAndWait(); txManager.startLong(); // shutdown to force a snapshot txManager.stopAndWait(); TransactionStateStorage txStorage = injector.getInstance(TransactionStateStorage.class); txStorage.startAndWait(); // confirm that the in-progress entry is missing a type TransactionSnapshot snapshot = txStorage.getLatestSnapshot(); assertNotNull(snapshot); assertEquals(1, snapshot.getInProgress().size()); Map.Entry<Long, TransactionManager.InProgressTx> entry = snapshot.getInProgress().entrySet().iterator().next(); assertNull(entry.getValue().getType()); // start a new Tx manager to test fixup Configuration conf2 = new Configuration(); // make sure we work with the default CDAP conf for snapshot codecs CConfiguration cconf = CConfiguration.create(); CConfigurationUtil.copyTxProperties(cconf, conf2); // override snapshot dir conf2.set(TxConstants.Manager.CFG_TX_SNAPSHOT_LOCAL_DIR, testDir.getAbsolutePath()); Injector injector2 = Guice.createInjector( new ConfigModule(conf2), new DiscoveryModules().getSingleNodeModules(), new TransactionModules().getSingleNodeModules()); TransactionManager txManager2 = injector2.getInstance(TransactionManager.class); txManager2.startAndWait(); // state should be recovered TransactionSnapshot snapshot2 = txManager2.getCurrentState(); assertEquals(1, snapshot2.getInProgress().size()); Map.Entry<Long, TransactionManager.InProgressTx> inProgressTx = snapshot2.getInProgress().entrySet().iterator().next(); assertEquals(TransactionType.LONG, inProgressTx.getValue().getType()); // save a new snapshot txManager2.stopAndWait(); TransactionStateStorage txStorage2 = injector2.getInstance(TransactionStateStorage.class); txStorage2.startAndWait(); TransactionSnapshot snapshot3 = txStorage2.getLatestSnapshot(); // full snapshot should have deserialized correctly without any fixups assertEquals(snapshot2.getInProgress(), snapshot3.getInProgress()); assertEquals(snapshot2, snapshot3); }
/** * In-progress LONG transactions written with DefaultSnapshotCodec will not have the type * serialized as part of the data. Since these transactions also contain a non-negative * expiration, we need to ensure we reset the type correctly when the snapshot is loaded. */ @Test public void testV2ToTephraV3Compatibility() throws Exception { long now = System.currentTimeMillis(); long nowWritePointer = now * TxConstants.MAX_TX_PER_MS; /* * Snapshot consisting of transactions at: */ long tInvalid = nowWritePointer - 5; // t1 - invalid long readPtr = nowWritePointer - 4; // t2 - here and earlier committed long tLong = nowWritePointer - 3; // t3 - in-progress LONG long tCommitted = nowWritePointer - 2; // t4 - committed, changeset (r1, r2) long tShort = nowWritePointer - 1; // t5 - in-progress SHORT, canCommit called, changeset (r3, r4) TreeMap<Long, TransactionManager.InProgressTx> inProgress = Maps.newTreeMap( ImmutableSortedMap.of( tLong, new TransactionManager.InProgressTx( readPtr, TransactionManager.getTxExpirationFromWritePointer( tLong, TxConstants.Manager.DEFAULT_TX_LONG_TIMEOUT), TransactionType.LONG), tShort, new TransactionManager.InProgressTx( readPtr, now + 1000, TransactionType.SHORT))); TransactionSnapshot snapshot = new TransactionSnapshot( now, readPtr, nowWritePointer, Lists.newArrayList(tInvalid), // invalid inProgress, ImmutableMap.<Long, Set<ChangeId>>of( tShort, Sets.newHashSet( new ChangeId(new byte[] {'r', '3'}), new ChangeId(new byte[] {'r', '4'}))), ImmutableMap.<Long, Set<ChangeId>>of( tCommitted, Sets.newHashSet( new ChangeId(new byte[] {'r', '1'}), new ChangeId(new byte[] {'r', '2'})))); Configuration conf1 = new Configuration(); conf1.set(TxConstants.Persist.CFG_TX_SNAPHOT_CODEC_CLASSES, SnapshotCodecV2.class.getName()); SnapshotCodecProvider provider1 = new SnapshotCodecProvider(conf1); ByteArrayOutputStream out = new ByteArrayOutputStream(); try { provider1.encode(out, snapshot); } finally { out.close(); } TransactionSnapshot snapshot2 = provider1.decode(new ByteArrayInputStream(out.toByteArray())); assertEquals(snapshot.getReadPointer(), snapshot2.getReadPointer()); assertEquals(snapshot.getWritePointer(), snapshot2.getWritePointer()); assertEquals(snapshot.getInvalid(), snapshot2.getInvalid()); // in-progress transactions will have missing types assertNotEquals(snapshot.getInProgress(), snapshot2.getInProgress()); assertEquals(snapshot.getCommittingChangeSets(), snapshot2.getCommittingChangeSets()); assertEquals(snapshot.getCommittedChangeSets(), snapshot2.getCommittedChangeSets()); // after fixing in-progress, full snapshot should match Map<Long, TransactionManager.InProgressTx> fixedInProgress = TransactionManager.txnBackwardsCompatCheck( TxConstants.Manager.DEFAULT_TX_LONG_TIMEOUT, 10000L, snapshot2.getInProgress()); assertEquals(snapshot.getInProgress(), fixedInProgress); assertEquals(snapshot, snapshot2); }
public static void stopServices() throws Exception { notificationService.stopAndWait(); datasetService.stopAndWait(); dsOpService.stopAndWait(); txManager.stopAndWait(); }